Live data from Hacker News

Titan: A Highly Scalable, Distributed Graph Database

thinkaurelius.github.com

11–20 of 41 posts

Re: Titan: A Highly Scalable, Distributed Graph Database

#11

Where's the SPARQL support?

Titan currently does not have "edge indexing" and thus can not implement Blueprints' GraphSail interface [https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementa...]. If it did implement GraphSail, then SPARQL would be supported via the Sail SPARQL engine.

Re: Titan: A Highly Scalable, Distributed Graph Database

#12
post #10
post #9

Earlier quoted context omitted.

Each Neo4j node stores all of the data, and it doesn't scale write horizontally well. OrientDB tries to scale writes (I'll be testing this in a few months), but still stores all of the data everywhere. This looks like it shards the data automagically. If it works well, I might be able to bang on it a bit, but I'm guessing that it gives shit performance for complex graph questions.

Titan exposes graph data over a machine cluster. It is an OLTP system that allows you to do local neighborhood graph traversals in sub-second time. For OLAP processing (e.g. global graph algorithms), Aurelius will be releasing two projects named Faunus and Fulgora in the coming months. These provide Hadoop connectivity and compressed in-memory representations of "graph slices." We will be publishing our talk slides t…

Congrats on the release, Marko!

Re: Titan: A Highly Scalable, Distributed Graph Database

#13
post #9
post #2

Interesting, does anyone know how this compares with the popular Neo4j Database?

Each Neo4j node stores all of the data, and it doesn't scale write horizontally well. OrientDB tries to scale writes (I'll be testing this in a few months), but still stores all of the data everywhere. This looks like it shards the data automagically. If it works well, I might be able to bang on it a bit, but I'm guessing that it gives shit performance for complex graph questions.

Also, Aurelius will be posting a blog post in ~2 weeks where we will be simulating Twitter. We replay Twitter from day 1 to June 2009 and slam it with ~10,000 concurrent users writing/reading follows relationships, tweets, stream constructions -- ultimately growing (what we think will be) a 3 billion edge graph when the simulation is complete.

Re: Titan: A Highly Scalable, Distributed Graph Database

#14

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site sugges…

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

Re: Titan: A Highly Scalable, Distributed Graph Database

#15
We are going to address alot of these questions in our presentation tonight. However, for the Hacker News crew that won't be there tonight, here is an early release of the Titan talk.

https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-grap...

http://titanbiggraphdata.eventbrite.com/

Enjoy!, Marko.

Re: Titan: A Highly Scalable, Distributed Graph Database

#16
post #10
post #9

Earlier quoted context omitted.

Each Neo4j node stores all of the data, and it doesn't scale write horizontally well. OrientDB tries to scale writes (I'll be testing this in a few months), but still stores all of the data everywhere. This looks like it shards the data automagically. If it works well, I might be able to bang on it a bit, but I'm guessing that it gives shit performance for complex graph questions.

Titan exposes graph data over a machine cluster. It is an OLTP system that allows you to do local neighborhood graph traversals in sub-second time. For OLAP processing (e.g. global graph algorithms), Aurelius will be releasing two projects named Faunus and Fulgora in the coming months. These provide Hadoop connectivity and compressed in-memory representations of "graph slices." We will be publishing our talk slides t…

I really wish that I could attend. We ask a lot of questions like:

Find all Nodes with a property in a tree

Find all leaves L of those nodes

Find all annotations in a DAG of those leaves

Collapse similar DAG entries by backtracking up the graph based on edge weights

Writes are bulk loaded, and right now, we are just trying to push all of the graph stuff offline, but there are some limitations to that, and we could really up our accuracy by being able to perform these queries quickly.

Re: Titan: A Highly Scalable, Distributed Graph Database

#17
post #14

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site sugges…

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

Worth a re-read: http://codahale.com/you-cant-sacrifice-partition-tolerance/

Re: Titan: A Highly Scalable, Distributed Graph Database

#18
post #14

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site sugges…

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

See https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-grap...

Re: Titan: A Highly Scalable, Distributed Graph Database

#19

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site sugges…

You could have said the same thing about generalized distributed computing as well. Much like with MapReduce, the trick is to recognize that there is an 80% solution that works quite well (based around BSP):

http://kowshik.github.com/JPregel/pregel_paper.pdf

http://googleresearch.blogspot.com/2009/06/large-scale-graph...

The trick is not to build a generalized graph operator solution. It's to have a specialized graph operator solution, and then see how many solutions you can fit to the specialized graph operators. Turns out you can do a lot with a little.

Re: Titan: A Highly Scalable, Distributed Graph Database

#20
post #14

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site sugges…

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

No, I am referring to the set of problems related to graph partitioning.

This is essentially the same underlying problem that is the source of why distributed NoSQL databases do not support join operations. In the case of NoSQL databases, they simply do not support joins because it is not a core operations. (Technically you can still do a join, it just has terrible scaling characteristics.)

The fundamental operation of graph databases are relational joins by another name, which means that graph databases have the same limitation on distribution that distributed NoSQL databases have on joins. However, unlike NoSQL databases it is their primary operation so they can't just not support it. Consequently, the only way to have a "graph database" that is massively distributable is to solve the same problem that prevents distributed databases from supporting joins.

Post reply on HN