Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Graph Databases 101

cray.com

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Graph Databases 101

#71

Post by akarambir »

rspeer wrote: post #63
grandalf wrote: post #61

Earlier quoted context omitted.

Explain

ConceptNet [1] started out as an academic project that I was responsible for for a while. Since then I've left to start a company but I still maintain ConceptNet and build lots of stuff on it. [1] http://conceptnet5.media.mit.edu Here's a list of databases, some of them graph databases, some of them barely databases, where I've tried to store and look up edges of ConceptNet: - SQLite - PostgreSQL - MongoDB - Some awf…

Can you explain how you were using PostgreSQL as Graph Database?

Re: Graph Databases 101

#72

Post by sklogic »

thesz wrote: post #59
sklogic wrote: post #33

Earlier quoted context omitted.

Of course you can express anything on top of a relational model. But for graphs such a representation would have been awfully inefficient. For this reason, CADs never even tried to switch to a relational data storage once that fancy new relational databases appeared, most of the professional CADs are still using good old graph databases.

I beg to disagree. I am part of the team developing Russian CAD system [0]. It uses what one can consider a hypergraph db (relation includes many objects), but that DBMS system has queries on par with SQL. And they prove themselves very useful in development of CAD. What you describe can be explained with development inertia. Most CADs are C/C++ and these languages are not very well suited for changes that go through…

I worked on a (very old) ship-building and factory-building CAD, written mostly in PL/I and Fortran. It was built around a graph database. Throughout its long and turbulent history, there were many attempts of moving to a relational storage, every time resulting in orders of magnitude drop in performance.

Keep in mind that in such a CAD designs are huge. Think of an aircraft carrier scale of "huge". And it was designed when memory was very limited. Therefore, pretty much all the CAD operations depended on the database access.

So, nobody really cared about the queries, they were insignificant. What people cared about was:

* Performance of following an arc

* Transactions

* Data consistency

* Compactness of representation (remember, disk space is also a limited thing when you're building aircraft carriers).

* Nice API (even in a very limited language)

Re: Graph Databases 101

#74

Post by rfreytag »

valhalla wrote: post #67

If anyone's curious about Network Science/Graph Theory in general here's a free online textbook used by a grad student friend of mine http://barabasilab.neu.edu/networksciencebook/downlPDF.html

Blasted down arrow is so close to the up-arrow I clicked down when I meant up. Someone please cancel out my mistake.

Re: Graph Databases 101

#76

Post by amirouche »

I am huge fan a graph-y stuff. I did several iteration over a graph database written -- in Python -- using files, bsddb and right now wiredtiger. I also use Gremlin for querying. Have a look at the code https://github.com/amirouche/ajgudb.

Also, I made an hypergraphdb, atom-centered instead of hyperedge focused in Scheme https://github.com/amirouche/Culturia/blob/master/culturia/c....

Did you know that Gremlin, is only srfi-41 aka. stream API with a few graph centric helpers.

edit: it's srfi 41, http://srfi.schemers.org/srfi-41/srfi-41.html

Re: Graph Databases 101

#77

Post by tkinom »

sophacles wrote: post #58
thesz wrote: post #29

It introduces false dichotomy "graph vs relational". In fact, most (if not all) graph algorithms can be expressed using linear algebra (with specific addition and multiplication). And matrix multiplication is a select from two matrices, related with "where i=j" and aggregation over identical result coordinates. The selection of multiplication and addition operations can account for different "data stored in links and…

Strictly speaking yeah. Practically speaking: Not really true. Just because something can be done, doesn't mean it can be done easily or well. I've done a lot of work with relational databases, and I love them for a lot of data sets. But I also have done a lot of work with graph databases - and they make working with graph shaped data a pleasure. I could do a graph in SQL, it's even moderately straight-forward in pos…

Facebook has a very good paper descriptions on how they do graph on top of relational database. Google "facebook tao" for details.

I read that and implement my own version with SQL in I am curious what I might be missing with that approach as compare to a real graph database?

Re: Graph Databases 101

#78

Post by amirouche »

gtrubetskoy wrote: post #56

I have spent a lot of time figuring out how to deal with a large graph a couple of years ago. My conclusion - there will never be such a thing as a "graph database". There are many efforts in this area, someone here already mentioned SPARQL and RDF, you can google for "triple stores", etc. There are also large-scale graph processing tools on top of Hadoop such as Giraph or Graphx for Spark. For the particular project…

In my point of view, the fact that you can add an expert index very easily to a graph database written in a modern language (say no C/C++) makes it even easier to customize an existing graph database to suit your direct need. In turn, storage and runtime can be tunned more easily. Making so easy to have the performance you need. But at the end of the day not dealing with algreba is the best.

Re: Graph Databases 101

#79

Post by rspeer »

akarambir wrote: post #71
rspeer wrote: post #63

Earlier quoted context omitted.

ConceptNet [1] started out as an academic project that I was responsible for for a while. Since then I've left to start a company but I still maintain ConceptNet and build lots of stuff on it. [1] http://conceptnet5.media.mit.edu Here's a list of databases, some of them graph databases, some of them barely databases, where I've tried to store and look up edges of ConceptNet: - SQLite - PostgreSQL - MongoDB - Some awf…

Can you explain how you were using PostgreSQL as Graph Database?

I mean, I was putting a graph in PostgreSQL, I don't know if that makes it a "graph database". Table of nodes, table of edges.

Re: Graph Databases 101

#80

Post by neunhoef »

rspeer wrote: post #9
iod wrote: post #4

Earlier quoted context omitted.

ArangoDB is free open source multi model no-sql db that has decent¹ perfomance with graph support: https://www.arangodb.com ¹ https://www.arangodb.com/2015/10/benchmark-postgresql-mongod...

"The performance will suffer if the dataset is much bigger than the memory." That is a huge drawback when compared to relational databases. A good follow-up question would be: which open-source graph databases can reasonably import and store graph data that's not small -- that is, more data than fits in than RAM? Without proprietary extensions?

One of the developers of ArangoDB here.

Let me explain this quotation. When your graph data (including indices) do no longer fit into the RAM of a single server, you can either live with the higher latency of loading data from disk or you can use sharding, which will lead to communication and therefore slower traversals.

That does not mean that things stop working, but performance will be less good, you can no longer visit tens of millions of nodes per second in a traversal as in RAM on a single server.

If you actually only traverse a much smaller hot subgraph, I would probably go for the disk based single server approach.

If your graph has a natural known clustering, then an optimized sharding solution with fine tuned sharding keys us probably your best bet.

You can do all this with ArangoDB.

However, graph traversals vary greatly in many respects, and your mileage may vary accordingly, with any approach.

I would love to chat in more detail about your use case.

Post reply on HN