Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: Dgraph 1.1.1

jepsen.io

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Jepsen: Dgraph 1.1.1

#41

Post by mrjn »

kesor wrote: post #16
enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

I have been developing a POC using dgraph for the last several months. I can't really comment on its robustness at production load since it was only used for local development. But I can comment that getting the "right syntax" was at times extremely frustrating. It has a lot of "there is just one way to do it, and you have to spend a month reading our code to find it" kind of thing going on. It is definitely "beta" s…

We now have a dedicated technical writer on the team, whose sole job is to improve our technical documentation. If you have suggestions, would be great if you could jot them down at [1], so we can improve the documentation.

[1]: https://discuss.dgraph.io.

Re: Jepsen: Dgraph 1.1.1

#42

Post by mrjn »

maximente wrote: post #23
mrjn wrote: post #13

Earlier quoted context omitted.

> there is no way to tell dgraph what the UID should be. There is. You can lease UIDs from Zero, and do your own assignment. Look at /assign endpoint [1] > doing an XID->UID lookup in order to create edges. Also, you can use upserts to do an XID lookup, before creating a new node. Which is practically what other DBs do too. > there is a bulk loader but that's only available in HA mode Don't know what that means. Bulk…

thanks for the reply! i hope this comes across as non-critical feedback, but it'd be really, really nice to put that assign endpoint in some form or fashion in the Mutation documentation. it is completely absent from there, and i don't recall seeing it in the tour of dgraph either. furthermore, it's absent from the golang client. the documentation states: > It’s possible to interface with Dgraph directly via gRPC or…

No worries at all, I like to hear feedback from users, whether its positive or negative. Though, I also like to separate wheat from chaff, which is why I have suggestions / follow up questions, etc.

Assign endpoint is something that you can just do once. You could say, give me a million UIDs, and then use them however you want. You don't need to call it repeatedly.

Also, its an endpoint to Zero, not to Alpha. Zeros are not supposed to be directly talked to, in a running cluster. We're now doing work around exposing some of Zero endpoints via Alphas, in our GraphQL rewrite of the /admin endpoint. So, that might make it easier.

I think the consistent theme I'm hearing here is that our documentation isn't clear -- we aim to improve that. But, could use more critical, logical feedback / suggestions on our forum -- so please feel free to pitch in there.

Re: Jepsen: Dgraph 1.1.1

#43

Post by eternalban »

aphyr wrote: post #29
bullen wrote: post #15

Earlier quoted context omitted.

Did you have to pay Kyle for him to test your database?

Yep! Testing databases is my full time job. Vendors pay me for the work, and that means the Jepsen library, test harness for each database, and all the reports are free for everyone. :)

Loved the precise, concise, and complete technical expression in the Background section.

(Thank you for sharing your great work.)

Re: Jepsen: Dgraph 1.1.1

#44

Post by dang »

sorokod wrote: post #25
bullen wrote: post #15

Earlier quoted context omitted.

Did you have to pay Kyle for him to test your database?

You may find the Ethics [1] section useful. Also, are you an acquaintance of Kyle Kingsbury? Just curious as you refer to him by his first name. [1] https://jepsen.io/ethics

It's an internet convention to refer to people by their first names.

Re: Jepsen: Dgraph 1.1.1

#45

Post by jcims »

mrjn wrote: post #34
wiradikusuma wrote: post #26

I've been using Dgraph for over a year (on/off, it's a side project). I first saw Dgraph on HN. I thought Dgraph was going to be the "secret sauce" for my app after reading the list of features (maybe I was mesmerized by the cute mascot). Few months down the line, though, sometimes I question my decision and whether I should have used the good old PgSQL. Let me explain. 1. Coming from SQL and key-value NoSQL, Dgraph…

Hmm... That's probably not the testimonial I was hoping to get from a Dgraph user. Though, I can see part of your pain is because of the custom query language, GraphQL+-. We now offer standard official GraphQL compliance as well, which tackles a lot of these issues you ran into. 1 and 2. GraphQL is becoming very common, so plenty of resources. 3. GraphQL has many amazing editors. 4. GraphQL allows for lack of referen…

TBH I think most of those complaints are true of the graph database ecosystem in general. I haven't used Dgraph but I echo the same concerns from my experience.

Re: Jepsen: Dgraph 1.1.1

#46

Post by cube2222 »

mrjn wrote: post #36
cube2222 wrote: post #19

Haven't used DGraph itself but I've used the storage engine they built for it - Badger, an alternative to RocksDB better optimized for SSD's - in two projects. One was for event saving and retrieval, which was able to sustain a stable 60k writes /s with simultaneous 10k reads /s. It worked great overall, with stuff needing nontrivial tuning being 1. RAM usage 2. If you overwhelm it with writes it'll stall to keep up…

Love it! You should send a PR to add OctoSQL to the list of projects using Badger (GitHub README).

I'll certainly do when we actually release the version based on badger!

Re: Jepsen: Dgraph 1.1.1

#47

Post by wpietri »

aphyr wrote: post #31
wpietri wrote: post #5

There's a part I don't get here: "To store large datasets Dgraph shards the set of triples by attribute, breaks attributes into one or more tablets, and assigns each tablet to a group of nodes." But earlier, it says, "For convenience, Dgraph can also represent all triples associated with a given entity as a JSON object mapping attributes to values—where values are other entities, that entity’s attributes and values a…

The first sentence is describing how Dgraph shards data for storage; the second sentence discusses how data can be represented in the query API. You're right that this could lead to broad fanout, if users typically retrieved all attributes for a given UID. It also impacts the performance of joins: graph traversal across a single attribute is much faster when all those edges are on the same node, but graph traversal a…

Thanks! That's exactly what I was looking to know. And it couldn't come from a more trusted source.

Re: Jepsen: Dgraph 1.1.1

#48

Post by aphyr »

anonymousDan wrote: post #38
aphyr wrote: post #30

Earlier quoted context omitted.

Yes: the library is prominently linked on the home page, and there are deep links to the Dgraph test suite code throughout the report. Pretty much all of my work is OSS, and public release of test harness for each report is explicitly part of the Jepsen ethics policy. :) https://github.com/jepsen-io https://jepsen.io/ https://jepsen.io/ethics

Awesome! Quick question - how much of the test harness you use for each report is generic/reusable, and how much is system specific? I have my students implement various algorithms/systems in Elixir e.g. Raft/Paxos, various broadcast algs etc. It would be nice to have something both they and I could use to simulate network partitions etc.

Kinda depends on what you're doing, how much of Jepsen you're using, and how complex the system-specific code is. You can write a minimal Jepsen test in ~100 lines of code, if that's helpful. Jepsen and its main supporting libraries (Elle and Knossos) clock in at about 19K lines of code; a little over six years of full-time work.

For simulation testing, I'd suggest looking at Maelstrom, which uses Jepsen to provide a sort of workbench for writing toy Raft implementations in any language. You give it a binary which takes messages as JSON on STDIN and emits messages to STDOUT; it spawns a bunch of "nodes" (local processes) of that binary, connects them via a simulated network, generates pathological network behavior, simulates client requests, and verifies the resulting histories with Jepsen.

https://github.com/jepsen-io/maelstrom

Re: Jepsen: Dgraph 1.1.1

#49

Post by aphyr »

mrjn wrote: post #39
aphyr wrote: post #31

Earlier quoted context omitted.

The first sentence is describing how Dgraph shards data for storage; the second sentence discusses how data can be represented in the query API. You're right that this could lead to broad fanout, if users typically retrieved all attributes for a given UID. It also impacts the performance of joins: graph traversal across a single attribute is much faster when all those edges are on the same node, but graph traversal a…

> graph traversal across different attributes might pay a higher latency cost Those can be done concurrently if at the same query level, so not necessarily any slower. In other terms, the number of network calls required (in a sufficiently distributed cluster, where each predicate/attribute is on a different server), is proportional to the number of attributes asked for in the query, not the number of results (at any…

>> graph traversal across different attributes might pay a higher latency cost

> Those can be done concurrently if at the same query level, so not necessarily any slower.

An important clarification, yes! I should have made that more explicit. :)

Post reply on HN