The Distributed Graph Database Titan Provides Real-Time Big Graph Data

Right now Titan traversals run locally and communicate with the back-end Cassandra or HBase datastore via Thrift, but they are working on moving the traversal code inside the Cassandra/HBase JVM so you can run Gremlin traversals inside the distributed cluster rather than than pulling the elements over the wire.

See https://groups.google.com/d/msg/gremlin-users/6GYiHn3QR8g/81...

Titan is distributed and has much higher write performance (5000 transactions/sec). In the words of Matthias Broecheler, the creator of Titan, you would use Titan to implement a social network (https://twitter.com/MBroecheler/statuses/213350753031569409).

Titan has pluggable storage options: Cassandra, HBase, Berkeley DB, and some have been working on adapters for DynamoDB and App Engine Datastore. However, it's not clear how well App Engine and DynamoDB will work with the upcoming Titan changes which will move the traversal engine into the cluster's JVM.

Both Neo4j and Titan support Blueprints, and you can run Gremlin (https://github.com/tinkerpop/gremlin/wiki) on both. Marko Rodriguez (http://markorodriguez.com/), the creator of Gremlin, is one of the principals behind Titan.

See "The Benefits of Titan": https://github.com/thinkaurelius/titan/wiki/The-Benefits-of-...

UPDATE: The benchmark in this post shows write performance on standard EC2 machines to be:

"Titan's performance under normal (5,900 transactions per second) and peak (10,200 transactions per second) load..."

This looks like a very nice implementation of a graph database. However a 6-machine cluster barely qualifies as "distributed" for the purposes of a graph database. You will experience almost no sub-linearity at this scale no matter how poorly the algorithms distribute. I am not intimately familiar with the transactional implementation but the graph processing algorithms described in Titan are not significantly distributable.

For graph queries, linearity can be definitely achieved on parallel systems with tens of thousands of compute nodes using the state-of-the-art algorithms. However, Titan does not use any of those algorithms and will be very lucky to exhibit linearity at a couple dozen nodes. Not to take away from the implementation but people looking for a scalable graph database will not be satisfied with Titan.

As an aside, the best algorithms and data structures for doing massively parallel graph operations are not published but obviously exist. The fastest system in the Graph500 benchmark uses a half million cores on a trillion edge graph. That is a several order of magnitude gap between what the best open source systems can do and what systems developed by closed research organizations can do as published in public benchmarks.

(Disclosure: I invented one of the aforementioned families of massively parallel graph processing algorithms in 2009, and not the first. The published literature has not even caught up with the first generation of such algorithms. A significant portion of advanced computer science research is no longer published.)

Yes. The R&D was almost entirely funded by multiple private research organizations over several years at relatively high cost (and not always the ones you would think). As a practical matter, many algorithm patents have not been enforceable for years even though they are well-supported in international law. Consequently, almost everything related to massively parallel graph technologies is closely held as a trade secret. Academia is pretty far behind the state-of-the-art in this area.

I've done work in other algorithm areas where the situation is similar. It often costs millions of dollars to develop major computer science advances but the best way to recover that investment is to leverage it without publishing it. Reverse engineering a thoroughly obfuscated technology takes a lot longer than when someone publishes the blueprints.

I will add that a lot of these algorithms are not at all obvious until you understand how they work. While knowing something is possible helps, it still requires a fair amount of theoretical computer science cleverness.