11–20 of 66 posts
I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failure scenarios.
Let me know if you have any questions, I'm around to answer.
Can anyone share their production experience with Dgraph?
We are doing PoCs around it -- however the text search is not ready for prime-time. https://github.com/dgraph-io/dgraph/issues/5102
It's in our backlog to improve FTS drastically from where it stands today.
one thing i'm not a huge fan of is dgraph's UID model, which is effectively an auto-incrementing uint across the entire cluster. because it auto increments server side, it's non-deterministic; ingesting 10 nodes before 10 others means that the UIDs will change across despite being the same XID. there is a way to use "blank nodes" to link nodes and edges with non-int UIDs, but that is only per-mutation, not per-commit…
There is. You can lease UIDs from Zero, and do your own assignment. Look at /assign endpoint [1]
> doing an XID->UID lookup in order to create edges.
Also, you can use upserts to do an XID lookup, before creating a new node. Which is practically what other DBs do too.
> there is a bulk loader but that's only available in HA mode
Don't know what that means. Bulk Loader is a single process (not distributed), and can be used to bootstrap a Dgraph cluster. The cluster can be a 2-node cluster, or an HA cluster, that doesn't matter.
> where even smallish batches of 100 hit some unspecified gRPC limit
Never heard of that. Grpc does have a 4GB per message limit. But, I doubt you'd hit that with 100 records.
There's a part I don't get here: "To store large datasets Dgraph shards the set of triples by attribute, breaks attributes into one or more tablets, and assigns each tablet to a group of nodes." But earlier, it says, "For convenience, Dgraph can also represent all triples associated with a given entity as a JSON object mapping attributes to values—where values are other entities, that entity’s attributes and values a…
> Naively, I'd think that one would cluster by entity and what an entity is connected to
In a distributed system, that approach leads high-fanout and network broadcasts, which kills query latency. Ideally, you want to do a traversal / join in one network call (max), not more. Because of this design, Dgraph can execute arbitrary depth queries in a much faster way. More details are in [1].
Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failu…
Can anyone share their production experience with Dgraph?
But I can comment that getting the "right syntax" was at times extremely frustrating. It has a lot of "there is just one way to do it, and you have to spend a month reading our code to find it" kind of thing going on. It is definitely "beta" software in that regard, and the ease of use of its query language (languages) is abysmal. The documentation is also extremely confusing and incomplete, to say the least. Needs a lot more examples and a lot more "ways to skin a cat" than currently documented.
On a good note, the support provided via discuss.dgraph.io is really good, even though there are so many people struggling to make it do simple things - that support forum will likely be a place where the answer can be found, or someone can answer (rather quickly) with some help.
Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failu…
Did you have to pay Kyle for him to test your database?
Can anyone share their production experience with Dgraph?
Some constraints we have: * We ingest what some may consider a lot of data - on the order of terabytes a day. This can be 10's of thousands of writes per second.
* We need transactional logic.
* We want to analyze that data as it comes in, so think 10x reads for every write.
GraphDBs out there didn't seem like they would cut it. I eliminated almost every database due to:
* Bad licensing
* Incapable of scaling writes horizontally, or generally anding tons of rights
* No ACID transactions
Most graphdbs out there had at least two of these issues.
DGraph so far has worked really well. We aren't sending it the full load of data yet, so there's still a question around that write load, but at least it's designed for that, and initial numbers have been promising.
The fact that it's liberally licensed, has a really good pricing model, has strong community support, good docs, etc, has made me glad I chose it.
The roughest part is probably the query language, because it's bespoke and therefor ends up having weird unexpected behavior sometimes. Now that it supports GraphQL that should be less of an issue.
One was for event saving and retrieval, which was able to sustain a stable 60k writes /s with simultaneous 10k reads /s. It worked great overall, with stuff needing nontrivial tuning being 1. RAM usage 2. If you overwhelm it with writes it'll stall to keep up with level 0/level 1 compactions.
Another one is OctoSQL[1], where we're building exactly-once event-time based stream processing all around Badger. So far it was a breeze and I don't think we'd build it if not for Badger.
Overall, at least the storage engine they're using is awesome, and I can definitely recommend it!
Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failu…