Dgraph Blog

I've followed Dgraph off and on. I enjoy looking through their open sourced work. A lot of it is interesting.

Manish seems to be THAT person. The technically competent, unwavering to the technical vision, doesn't work well with others person at your company. He went on to found a startup so he could do what he wants, and unfortunately has now realized board members are very similar to managers at Google.

Here's a blog post where he stepped down from being CEO to being CTO: https://dgraph.io/blog/post/changing-guard/

> I’m very confident in my abilities to bring in a solid VP team and lead the GTM motion. But, I’ve also realized that Dgraph is an intrinsically technical product. While I can find other S&M folks, finding a true engineering replacement for me has been, and will continue to be a real challenge.

Reviews from Glassdoor confirm this inability to let others touch his baby: https://www.glassdoor.com/Reviews/Dgraph-Labs-Reviews-E1284992.htm

> Massive layoffs with no warning and almost no severance pay. This is a sinking ship, not a rocket ship - avoid it!
Inept founder (was CEO and CTO while I was there) who only believes in his ability and doesn't trust employee expertise. He would rather be right than make money or see his company succeed. Manish doesn't listen to employees or managers and even disputes data showing false assumptions or failed business approaches. Micromanages in departments where he has minimal knowledge or understanding, leading to low morale…

LMDB and RockDB are both great projects, but how much performance you can get out of them would depend on the larger architecture of the system they are being integrated into. Both projects provide tens of flags to adjust read/write amplification to customize them for a particular use case. You will often see public benchmarks of these systems being updated with suggestions from both sides on less-than-optimal configurations being used!

In the case of Typesense, RocksDB is not even a top-10 contributor to the overall latency involved in serving the result. In any case, it would be good to clarify a few things:

> As we don’t use RocksDB but LMDB, we use a lot less real memory than key-value stores that uses a user-side cache system.

Typesense stores only the raw-data in RocksDB. All indexing data structures for filtering, faceting etc. are compact in-memory data structures stored outside. The only fixed memory cost from RocksDB is an in-memory table that is used to buffer writes (see the next point) before being flushed to disk. In practice, this is a trivial percentage of memory used when compared to other data structures.

> LSM-Tree key-value store like RocksDB that needs to merge/compact pages of key-values pairs before being able to return it too you

This happens in-memory and is flushed to the disk in batches. Merging of on-disk SST files happens in the background with no real impact on reads. The advantage of this approach though is that it gets you really good…

Looks like the big challenge is managing a large, LRU cache, which tends to be a difficult problem for GC runtimes. I bet the JVM, with its myriad tunable GC algorithms, would perform better, especially Shenandoah and, of course, the Azul C4.

The JVM world tends to solve this problem by using off-heap caches. See Apache Ignite [0] or Ehcache [1].

I can't speak for how their Rust cache manages memory, but the thing to be careful of in non-GC runtimes (especially non-copying GC) is memory fragmentation.

Its worth mentioning that the Dgraph folks wrote a better Go cache [2] once they hit the limits of the usual Go caches.

From a purely architectural perspective, I would try to put cacheable material in something like memcache or redis, or one of the many distributed caches out there. But it might not be an option.

It's worth mentioning that Apache Cassandra itself uses an off-heap cache.

[0]: https://ignite.apache.org/arch/durablememory.html
[1]: https://www.ehcache.org/documentation/2.8/get-started/storage-options.html#bigmemory-(off-heap-store)
[2]: https://blog.dgraph.io/post/introducing-ristretto-high-perf-go-cache/

Dgraph Labs | San Francisco, CA or Bangalore, India | ONSITE

We announced our Series A funding this week[1]!

Dgraph is the world’s most advanced graph database. It is horizontally scalable, synchronously replicated, transactional and distributed. It can do arbitrarily deep joins, while minimizing network communication and disk seeks, scaling well as cluster size increases. It is a low-latency, high-throughput database, serving complex queries over multiple independent data sources in real-time[2].

Dgraph Labs was started by ex-Google engineers. We are a small team of dedicated engineers with a mission to build world’s best graph database. Nothing less excites us!

Open Positions:

- Quality Assurance Engineer
- DevOps Engineer
- Developer Advocate
- Distributed Systems Engineer
- Full-Stack Engineer

Apply here: https://dgraph.io/careers.

[1] https://blog.dgraph.io/post/how-dgraph-labs-raised-series-a/

[2] https://github.com/dgraph-io/dgraph

Dgraph Labs | Palo Alto, Bangalore, Remote | Hiring Community Engineers, Technical Writers, Distributed Systems, Frontend, and Fullstack Engineers | Full-time, VISA | https://dgraph.io/

Dgraph Labs was started by ex-Google engineers and is VC funded. We are a team of dedicated engineers with a mission to build the world's best graph database. Nothing less excites us! Dgraph has solid open-source traction with over 13,000 GitHub stars and over 2 million Docker Hub pulls and is being used in production at multiple Fortune 500 companies.

We are hiring for community engineers, frontend engineers, fullstack engineers, technical writers, and distributed systems engineers. We love what we do and take care of each other while hand-crafting the best graph database.

Stack: Go, ReactJS, Javascript, Docker, k8s

More details: https://jobs.lever.co/dgraph

If this sounds exciting, drop-in a mail to anurag@dgraph.io, also mention HN in the subject line.

I agree it seems like potentially the same problem, and it manifests in forcing devs to write 'resolvers' which looks like horrible drudge work. So I wonder if there are decent offerings without the impedance mismatch between GraphQL queries and the database (so we don't have to translate them into SQL). I did a very quick look and at least found this: https://dgraph.io/

I'm getting the impression that if nothing else though, I'm probably gonna need to wait a few years for this tech to stabilize more. Which I find very unfortunate, because the standard way of doings things these days feels very unpleasant to me (I hate writing boilerplate more than anything: have an overuse injury, so typing is the worst part of coding)—and the GraphQL queries do seem to make a lot of sense (although forcing clients to have explicit knowledge of schema structure seems a little dangerous...).