Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: Dgraph 1.1.1

jepsen.io

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Jepsen: Dgraph 1.1.1

#21

Post by kwesthaus »

enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

Wasn't at a huge scale but for one project another intern and I took a proof of concept that another engineer had done with Gremlin [1] and turned it into a full tool and ended up using dgraph. The python bindings were easy to work with and Ratel (the UI/web frontend) made quick searches and tests easy.

I liked working with it so now I'm the package maintainer for it on AUR [2]. At some point I'd like to make a repo showing how to implement common graph algorithms with the python bindings, since GraphQL+- currently only supports k-shortest path at the query level [3].

[1] https://tinkerpop.apache.org/gremlin.html [2] https://aur.archlinux.org/packages/dgraph-bin/ , https://aur.archlinux.org/packages/dgraph-git/ [3] https://discuss.dgraph.io/t/how-about-doing-some-graph-compu...

Re: Jepsen: Dgraph 1.1.1

#22

Post by mro_name »

mrjn wrote: post #11

Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failu…

Hi, was nosing around your website and find it very much appealing. Am currently for hire, rooted in germany. Whom should I contact? Thx! Am author of https://github.com/mro/librdf.sqlite

Re: Jepsen: Dgraph 1.1.1

#23

Post by maximente »

mrjn wrote: post #13
maximente wrote: post #6

one thing i'm not a huge fan of is dgraph's UID model, which is effectively an auto-incrementing uint across the entire cluster. because it auto increments server side, it's non-deterministic; ingesting 10 nodes before 10 others means that the UIDs will change across despite being the same XID. there is a way to use "blank nodes" to link nodes and edges with non-int UIDs, but that is only per-mutation, not per-commit…

> there is no way to tell dgraph what the UID should be. There is. You can lease UIDs from Zero, and do your own assignment. Look at /assign endpoint [1] > doing an XID->UID lookup in order to create edges. Also, you can use upserts to do an XID lookup, before creating a new node. Which is practically what other DBs do too. > there is a bulk loader but that's only available in HA mode Don't know what that means. Bulk…

thanks for the reply!

i hope this comes across as non-critical feedback, but it'd be really, really nice to put that assign endpoint in some form or fashion in the Mutation documentation. it is completely absent from there, and i don't recall seeing it in the tour of dgraph either.

furthermore, it's absent from the golang client. the documentation states:

> It’s possible to interface with Dgraph directly via gRPC or HTTP. However, if a client library exists for you language, this will be an easier option.

however it looks like i'll need an additional HTTP layer to interface with the /assign endpoint. not a huge deal, but that seems like a big functionality gap with the golang endpoint - would definitely like to see that added in there.

lastly, the /assign endpoint and the bulk loader can only be run with a DGraph Zero instance, which, as far as i can tell, which doesn't run by default with the provided docker image. that's an important detail that's not super duper obvious from the docs, until you start seeing parameters like dgraph-zero, and then realizing that it doesn't come with the quick start docker image.

again, hope this isn't taken personally. thanks for your work on the project!

Re: Jepsen: Dgraph 1.1.1

#24

Post by mavbo »

mrjn wrote: post #11

Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failu…

I've been using Dgraph in building some PoCs and it's been great. My current confusion is around the push for a plain GraphQL compliant query endpoint as I'm not sure what the intended purpose is. Presumably it's not for direct connections between a front-end web client and the DB using something like Apollo Client. Should front ends still be going through some sort of API server that ingests GraphQL or REST style requests and queries Dgraph to build results? That's what I'm currently doing to implement things such as custom logic, side effects, authentication, etc. I'm not seeing the benefit of the plain GraphQL endpoint in that case, since I would be managing a separate GraphQL schema in the DB that is likely a subset or is similar to the API server's GraphQL schema. I've been sticking with GraphQL+- since it works great and I need an API server in between the front end and DB anyways. I've built some helper functions to map plain GraphQL request trees into expanded GraphQL+- queries for expanding edges since my API schema closely matches my Dgraph schema.

Would love some insight into the plans or use case for the plain GraphQL endpoint.

Re: Jepsen: Dgraph 1.1.1

#25

Post by sorokod »

bullen wrote: post #15
mrjn wrote: post #11

Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failu…

Did you have to pay Kyle for him to test your database?

You may find the Ethics [1] section useful.

Also, are you an acquaintance of Kyle Kingsbury? Just curious as you refer to him by his first name.

[1] https://jepsen.io/ethics

Re: Jepsen: Dgraph 1.1.1

#26

Post by wiradikusuma »

I've been using Dgraph for over a year (on/off, it's a side project). I first saw Dgraph on HN.

I thought Dgraph was going to be the "secret sauce" for my app after reading the list of features (maybe I was mesmerized by the cute mascot). Few months down the line, though, sometimes I question my decision and whether I should have used the good old PgSQL. Let me explain.

1. Coming from SQL and key-value NoSQL, Dgraph was very foreign to me. It's like OOP guy learning Functional Programming. Now I'm quite comfortable with it, but it took me months to be productive. To give an analogy, learning Dgraph is like learning Scala instead of Python.

2. Actually it's worse, the resources are mostly in the forum (micheldiz is my hero) and the official website. Until recently, the navigation for the docs is horrible, everything in one big HTML file, difficult to jump in from Google search result.

3. Rudimentary tooling for development phase. When you're working on a new idea for a product, you will experiment A LOT with the schema and data. When you write wrong data, with most RDMBS you can use GUI to right-click and delete or edit. In Dgraph, you must write mutation query (assuming you remember the syntax, as it is a bespoke language). Dgraph GUI is very minimal.

4. About the mutation.. in Dgraph (as far as I know) there's no referential integrity in the DB-sense. Like, you can make FK to non-existent object, or insert something invalid without returning error (but it's not stored, since it's invalid). The "integrity check" is in your app.

5. Because of #3 and #4, I find it easier to just drop the whole database and recreate it along with seed data, every. time.

6. The documentation for the Java client library is very minimal. So there you go, unfamiliarity with the query language ("GraphQL+-"), with the Dgraph itself, and with the client library.

I still use Dgraph, it's a good fit for my app, but if you're starting on a new business idea, maybe don't use anything fancy. My mistake, being a developer, I mixed research (new tech!) and bootstrapping.

(In case you're wondering, my app is http://s.id/axtiva-android-test -- still version 0.0.x, but recently I'm releasing weekly)

Re: Jepsen: Dgraph 1.1.1

#27

Post by anonymousDan »

Does Jepsen ever open source any of their tooling/test harnesses? I teach a distributed systems class and it would be great to have an automated test framework/tools for Distributed systems issues

Re: Jepsen: Dgraph 1.1.1

#28

Post by onefuncman »

mrjn wrote: post #12
chintan wrote: post #7

Earlier quoted context omitted.

We are doing PoCs around it -- however the text search is not ready for prime-time. https://github.com/dgraph-io/dgraph/issues/5102

(author of Dgraph) We want to improve full text search, to bring it inline with Elastic Search. A lot of people compare Dgraph against Elastic, because they'd rather just have one solution (Dgraph) instead of two. It's in our backlog to improve FTS drastically from where it stands today.

do you have any reusability of the infrastructure for indexing edge properties to reuse in FTS?

Re: Jepsen: Dgraph 1.1.1

#29

Post by aphyr »

bullen wrote: post #15
mrjn wrote: post #11

Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failu…

Did you have to pay Kyle for him to test your database?

Yep! Testing databases is my full time job. Vendors pay me for the work, and that means the Jepsen library, test harness for each database, and all the reports are free for everyone. :)

Re: Jepsen: Dgraph 1.1.1

#30

Post by aphyr »

anonymousDan wrote: post #27

Does Jepsen ever open source any of their tooling/test harnesses? I teach a distributed systems class and it would be great to have an automated test framework/tools for Distributed systems issues

Yes: the library is prominently linked on the home page, and there are deep links to the Dgraph test suite code throughout the report. Pretty much all of my work is OSS, and public release of test harness for each report is explicitly part of the Jepsen ethics policy. :)

https://github.com/jepsen-io https://jepsen.io/ https://jepsen.io/ethics

Post reply on HN