Live data from Hacker News

Titan: A Highly Scalable, Distributed Graph Database

thinkaurelius.github.com

21–30 of 41 posts

Re: Titan: A Highly Scalable, Distributed Graph Database

#21
post #19

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site sugges…

You could have said the same thing about generalized distributed computing as well. Much like with MapReduce, the trick is to recognize that there is an 80% solution that works quite well (based around BSP): http://kowshik.github.com/JPregel/pregel_paper.pdf http://googleresearch.blogspot.com/2009/06/large-scale-graph... The trick is not to build a generalized graph operator solution. It's to have a specialized graph…

Approaches like Pregel have been the canonical way of dealing with large graph problems for many years. Unfortunately, most interesting graph analytic problems do not fit into that model because the "graph-like" aspect is still limited to problem sizes that fit conventional algorithms.

For example, if you can reduce a trillion-edge graph analysis problem into a billion-edge graph plus some other stuff (usually materialized document structures) then you can fit that into something like Pregel. That is how almost all real-world graph analysis is done today.

But by doing so, you've lost the ability to do graph analysis on the other ~trillion edges for the sake of tractability in a narrow case. You can't do relationship analysis across the attached documents. There are many, many graph analytic problems that require a true graph that is orders of magnitude larger than what can be partitioned even after accounting for graph reduction techniques such as those used in Pregel.

The Holy Grail is still the ability to run ad hoc graph analytic queries against a massively distributed graph representation. There are no shortcuts around this for many interesting applications. Right now, we are limited to mere billions of edges for most practical purposes and all of the hacks and workarounds are designed to keep the number of true edges to around this number even when the data model is much larger.

Re: Titan: A Highly Scalable, Distributed Graph Database

#22
post #14

Earlier quoted context omitted.

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

No, I am referring to the set of problems related to graph partitioning. This is essentially the same underlying problem that is the source of why distributed NoSQL databases do not support join operations. In the case of NoSQL databases, they simply do not support joins because it is not a core operations. (Technically you can still do a join, it just has terrible scaling characteristics.) The fundamental operation…

There are several distributed triple stores (triple stores are a specialised graph store).

http://4store.org http://virtuoso.openlinksw.com/ http://www.systap.com/bigdata.htm

There's a paper on 4store here: http://4store.org/publications/harris-ssws09.pdf

Re: Titan: A Highly Scalable, Distributed Graph Database

#23
post #6

Also, What is interesting about a graph database relative to simple key-value database? Storing edges of a graph is trivial for a key-value store and so it seems like any key-value store could let store the basic graph structure? Do graph databases support graphic-specific queries and indices?

I can only answer for one advantage I specifically know of regarding graph DB's over key value: dynamic, mergeable schemas which enforce data integrity WITHIN the database rather than with code on top of it. There are many, many people on HN who are much more knowledgeable than I am on graph DB's, and I sure as hell hope they answer on this question. I'm curious if this supports the RDF, OWL, and SPARQL standards? I'…

It looks like it's Gremlin only.

What triple stores have you looked at? 4store is performant, but doesn't support reasoning. There's also BigData and Virtuoso which support various levels of it, and Franz are apparently working on a clustered version of Allegrograph.

Re: Titan: A Highly Scalable, Distributed Graph Database

#24
post #14

Earlier quoted context omitted.

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

No, I am referring to the set of problems related to graph partitioning. This is essentially the same underlying problem that is the source of why distributed NoSQL databases do not support join operations. In the case of NoSQL databases, they simply do not support joins because it is not a core operations. (Technically you can still do a join, it just has terrible scaling characteristics.) The fundamental operation…

[deleted]

Re: Titan: A Highly Scalable, Distributed Graph Database

#25

Also, What is interesting about a graph database relative to simple key-value database? Storing edges of a graph is trivial for a key-value store and so it seems like any key-value store could let store the basic graph structure? Do graph databases support graphic-specific queries and indices?

Sure you can easily store graphs in a KV store, but graph databases allow you to efficiently traverse the graph too. Look up how to traverse graph structures in hadoop for example - I was looking it up recently for something Im building and in the end I opted for using hadoop to filter, aggregate and format the raw data and then input the preprocessed data into graph databases (Im using orientdb at the moment) for traversal queries. I duplicate the data on multiple machines and use storm bolts to do geaph queries (so each storm bolt has its own local copy of the graph database). Seems to be working well so far and allows me to do realtime traversals (though inserts wait hadoop the next hadoop batch job). Seemed a lot easier than what Id have to do to efficiently process using hadoop.

Re: Titan: A Highly Scalable, Distributed Graph Database

#26
post #18
post #14

Earlier quoted context omitted.

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

See https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-grap...

Thank you, this shows that for the one you want, the choice is in the backend.

Re: Titan: A Highly Scalable, Distributed Graph Database

#27
post #17
post #14

Earlier quoted context omitted.

You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.

Worth a re-read: http://codahale.com/you-cant-sacrifice-partition-tolerance/

Thank you, this is really great to show AP or CP but never CA and call it distributed.

Re: Titan: A Highly Scalable, Distributed Graph Database

#28

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site sugges…

Agreed, this is why our solution was to use Node.JS and achieve parallelism through eventually consistent independent data-stores separated on each physical web-server application instance, I'll gladly switch to a distributed centrally managed system when it becomes available.

Re: Titan: A Highly Scalable, Distributed Graph Database

#29
I don't want to spam this thread, but I feel I have to plug my personal pet project Related (https://github.com/sutajio/related), since it is, well.. yeah... "related" to the topic being discussed.

It can't do half of what Titan does and is a much, much simpler design. But it is fast, easy to use and works really well for 80% of the use cases you might have for a graph database on the web (social graphs, semantic web stuff, etc.)

Re: Titan: A Highly Scalable, Distributed Graph Database

#30

Also, What is interesting about a graph database relative to simple key-value database? Storing edges of a graph is trivial for a key-value store and so it seems like any key-value store could let store the basic graph structure? Do graph databases support graphic-specific queries and indices?

What's interesting about key-value databases? Storing keys and values in an RDBMS is trivial...

Graph databases are great for datasets where structure matters more than in a relational database and (way) more than in a K/V-store. When you're doing traversals of arbitrary depth, other technologies fall on their collective face.

Post reply on HN