Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: Dgraph 1.1.1

jepsen.io

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Jepsen: Dgraph 1.1.1

#51

Post by lmeyerov »

chintan wrote: post #7
enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

We are doing PoCs around it -- however the text search is not ready for prime-time. https://github.com/dgraph-io/dgraph/issues/5102

FWIW, we do a lot of 'GPU visual graph analytics and investigation for X' work at Graphistry, where X is hooking into either graph DBs (neo4j, ...) or doing as a virtual / on-the-fly layer over other data systems (Splunk, jupyter notebooks, ...). Almost all of our user's graph projects have ended up involving text search, and as part of that, search indexes. Think security, fraud, genetics, etc. I can only think of a few exceptions that did not need text, such as blockchain viz. I just sort of assume text fields as part of linking data nowadays. In fact, a lot of our recent work is going to the next level, where we use ML algs to compute over text to infer even fuzzier connections, vs simple ID/string/regex matching from the older days of graph tech.

So at least for domains where people want to make correlations over data such as a logs, events, transactions, CSVs, etc., I encourage dgraph folks to watch discussions of text closely.

Fun recent example that illustrates this: For ProjectDomino.org (COVID anti-misinfo), we started by ingesting the covid twitter firehose into a graphdb for easy and fast pivoting by tweet/account/etc. However, our analysts need to search by text, and a lot of our current work is now doing ML/graph algorithms to mine the text to infer fuzzy edges: GPU BERT, GPU UMAP, ... . Neo4j supports setting up various text indexes which helps search, but for analytics, we end up having to extract the data out of the DB, infer relationships & scores, and put them back in.

Re: Jepsen: Dgraph 1.1.1

#52

Post by mrjn »

jcims wrote: post #45
mrjn wrote: post #34

Earlier quoted context omitted.

Hmm... That's probably not the testimonial I was hoping to get from a Dgraph user. Though, I can see part of your pain is because of the custom query language, GraphQL+-. We now offer standard official GraphQL compliance as well, which tackles a lot of these issues you ran into. 1 and 2. GraphQL is becoming very common, so plenty of resources. 3. GraphQL has many amazing editors. 4. GraphQL allows for lack of referen…

TBH I think most of those complaints are true of the graph database ecosystem in general. I haven't used Dgraph but I echo the same concerns from my experience.

Sentiments are changing now, particularly with GraphQL. Dgraph bet on GraphQL early on, and it's really catching on as a replacement for REST.

Re: Jepsen: Dgraph 1.1.1

#53

Post by j-e-k »

johnymontana wrote: post #9
simonw wrote: post #2

I find Dgraph to be one of the most interesting of the current batch of non-relational data stores. I've wanted a robust, easy-to-use graph database for years (ever since reading about the crazy brilliant graph database stuff that goes on inside Facebook https://www.facebook.com/notes/facebook-engineering/tao-the-... ) and Neo4J never really cut it for me. Watching Dgraph mature - and survive two rounds of Jepsen wit…

> Neo4J never really cut it for me. What did you find lacking in Neo4j?

In my experience, from ~1.5 years ago, performance became an extreme challenge to overcome if you wanted to service user facing requests in <100ms (for a non-trivial sized graph). I really did enjoy Cypher though and the tooling around it was very polished. Tempted to try it again now that they have a new version.

Re: Jepsen: Dgraph 1.1.1

#54

Post by simonw »

johnymontana wrote: post #9
simonw wrote: post #2

I find Dgraph to be one of the most interesting of the current batch of non-relational data stores. I've wanted a robust, easy-to-use graph database for years (ever since reading about the crazy brilliant graph database stuff that goes on inside Facebook https://www.facebook.com/notes/facebook-engineering/tao-the-... ) and Neo4J never really cut it for me. Watching Dgraph mature - and survive two rounds of Jepsen wit…

> Neo4J never really cut it for me. What did you find lacking in Neo4j?

Last time I looked at it it seemed to really want me to use Java.

Turns out the official Python client library is five years old now so I clearly need to update my mental model of what it can do!

Re: Jepsen: Dgraph 1.1.1

#55

Post by abc-xyz »

enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

I love working with Dgraph but I hate DBA-related work. As a result I only use it for local projects. I’ve once experienced the data becoming inaccessible and being unable to restart the docker containers (which sadly happened before I began to regularly export data, but luckily with data that wasn’t too important), but I’ve otherwise only had positive experiences (micheldiz is great).

I’d probably use it for pet projects if they made it easier to automate backups to the cloud, and I’d use it for all projects if they offered a hosted solution.

Re: Jepsen: Dgraph 1.1.1

#56

Post by mrjn »

abc-xyz wrote: post #55
enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

I love working with Dgraph but I hate DBA-related work. As a result I only use it for local projects. I’ve once experienced the data becoming inaccessible and being unable to restart the docker containers (which sadly happened before I began to regularly export data, but luckily with data that wasn’t too important), but I’ve otherwise only had positive experiences (micheldiz is great). I’d probably use it for pet pro…

Something is in the works around a managed service!

Re: Jepsen: Dgraph 1.1.1

#57

Post by pfhorge »

enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

Somebody in my organization got very interested in Dgraph in its 0.7 or 0.8 days. The version was marked as "production ready", but it was an absolute trainwreck.

We were modelling individuals and contacts between them, and the cluster would constantly break with dataset sizes that should have been easily managed. There was clearly something wrong in the storage engine, because we saw insane disk space usage. Dgraph consumed 10s of TB for something that should have taken We were one of the largest installations at the time, and were working with the core development team, but they were never able to resolve the issues.

We eventually had to tell management that there was no way we'd be able to operate the thing given its disk space consumption rate, so we had to delay project delivery to rip out Dgraph and replace it with Postgres.

Surely it's better today, but I'll never use it again by choice.

Re: Jepsen: Dgraph 1.1.1

#58

Post by freelancercv »

pfhorge wrote: post #57
enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

Somebody in my organization got very interested in Dgraph in its 0.7 or 0.8 days. The version was marked as "production ready", but it was an absolute trainwreck. We were modelling individuals and contacts between them, and the cluster would constantly break with dataset sizes that should have been easily managed. There was clearly something wrong in the storage engine, because we saw insane disk space usage. Dgraph…

I don't agree with this high size database gap.

Dgraph is built for performance and with one of our app, we faced similar challenges. After reading some documentation and watching some of their videos, I got to know they compromise space against performance when we have lots of index. We reduced index from 35 to 8 and the db size got drastically low.

I believe you should investigate that as well to check if it's wrong in your database architecture design.

For your gap of <100 GB against 10,000 GB. I assume, you probably created lots of index. Just create good database design and reduce index, you will have low size high performance app.

Re: Jepsen: Dgraph 1.1.1

#59

Post by mrjn »

pfhorge wrote: post #57
enzotar wrote: post #3

Can anyone share their production experience with Dgraph?

Somebody in my organization got very interested in Dgraph in its 0.7 or 0.8 days. The version was marked as "production ready", but it was an absolute trainwreck. We were modelling individuals and contacts between them, and the cluster would constantly break with dataset sizes that should have been easily managed. There was clearly something wrong in the storage engine, because we saw insane disk space usage. Dgraph…

v0.7? That was Dec 2016. A lot has changed in 3.5 years.

Re: Jepsen: Dgraph 1.1.1

#60

Post by kwesthaus »

mrjn wrote: post #37
kwesthaus wrote: post #21

Earlier quoted context omitted.

Wasn't at a huge scale but for one project another intern and I took a proof of concept that another engineer had done with Gremlin [1] and turned it into a full tool and ended up using dgraph. The python bindings were easy to work with and Ratel (the UI/web frontend) made quick searches and tests easy. I liked working with it so now I'm the package maintainer for it on AUR [2]. At some point I'd like to make a repo…

Thanks for maintaining Dgraph on AUR. I'm a fan of Arch Linux. I think the latest release is 20.03.1, perhaps time to update?

Sure thing!

Will do, I've been a little lax the past few weeks but my school semester just finished so now I should have more time again.

Post reply on HN