Live data from Hacker News

The Distributed Graph Database Titan Provides Real-Time Big Graph Data

thinkaurelius.com

11–20 of 21 posts

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#11
post #9

Earlier quoted context omitted.

Why is it that they aren't published? Competitive advantage?

Yes. The R&D was almost entirely funded by multiple private research organizations over several years at relatively high cost (and not always the ones you would think). As a practical matter, many algorithm patents have not been enforceable for years even though they are well-supported in international law. Consequently, almost everything related to massively parallel graph technologies is closely held as a trade sec…

I understand you may be under NDA, as well as the general pressure to not reveal knowledge that is the basis of your career, but would you care to point out any open literature you think is worthwhile to study?

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#12

Great article. I'd like to see a few other details that aren't mentioned: - What's the following distribution end up looking like? Does it have a similar fraction of 'celebrity' users with huge follower counts? Or more technically, does the russian roulette against the recommendation sampling end up producing a network similar to a scale free graph grown via preferential attachment (Barabási–Albert model)? It looks l…

Hey,

- the data we used was crawled by Kwak et. al in 2009. We wanted to use a real social network dataset for the experiment and that was the largest/most useful one we could find. Other than de-duplication we did not make any modifications to the dataset, so the statistics reported in their paper still hold: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.153....

- You mean what is the overhead induced by pre-computing the stream edge rather than collecting the relevant streams at query time? You are right that this requires a significant amount of storage space, however, as you also pointed out, this will get cold quickly and be sitting on disk only (i.e. not taking up space in the valuable cache). The reason this is very efficient is because of the time-based vertex centric index we build for the stream edges. This allows us to quickly pull out the most recent tweets for any user. If we had to compute those at query time, we would have to traverse to each person followed, get their 10 most recent tweets and then merge those in-memory. That would be significantly more expensive and since stream reading is probably the most frequent activity on twitter, pre-computing it saves a lot of time at the expense of inexpensive disk storage.

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#13

Cassandra is turning out to be one killer data backend. It's really exciting to see what's being built on top of it. Hadoop, Solr/Lucene, and now Blueprints/Grapb DB operations are all available on the same Cassandra cluster, in addition to the stuff Cassandra does quote-unquote natively. Add Zookeeper for the few times you need an honest-to-goodness transaction and it's just crazy how good the tech has gotten on the…

Absolutely, without NoSQL solutions like Cassandra Titan would not be possible.

Regarding Zookeeper: We actually build a locking system into Titan that uses quorum reads/writes with time-outs and cleanup to ensure consistency for certain edge/property types as defined inside Titan. This gives you consistency guarantees out of the box without having to introduce another component (like Zookeeper) into your deployment. For infrequent lock usage (which I strongly encourage ;-) this should be sufficient. For frequent locking, something like Zookeeper is far superior.

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#14

This looks like a very nice implementation of a graph database. However a 6-machine cluster barely qualifies as "distributed" for the purposes of a graph database. You will experience almost no sub-linearity at this scale no matter how poorly the algorithms distribute. I am not intimately familiar with the transactional implementation but the graph processing algorithms described in Titan are not significantly distri…

> people looking for a scalable graph database will not be satisfied with Titan.

Are there any open source graph databases that are scalable in your opinion? Which one would you recommend?

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#15

Earlier quoted context omitted.

Yes. The R&D was almost entirely funded by multiple private research organizations over several years at relatively high cost (and not always the ones you would think). As a practical matter, many algorithm patents have not been enforceable for years even though they are well-supported in international law. Consequently, almost everything related to massively parallel graph technologies is closely held as a trade sec…

I understand you may be under NDA, as well as the general pressure to not reveal knowledge that is the basis of your career, but would you care to point out any open literature you think is worthwhile to study?

For massively distributed and parallel algorithms, a core part of the computer science is based around space decomposition data structures. If your algorithm is not based on a space decomposition structure, it won't linearly parallelize across a vast number of cores. When I say some algorithm won't distribute or parallelize, this is what I am usually (but not always) looking at.

Space decomposition structures take a metric space and decompose it using an invariant space partitioning function. The partition points are fixed regardless of data distribution, the only decision process is when to apply the partitioning function.

The two well-known examples in literature are quad-trees and distributed hash tables. These are both very simple, narrow examples of space decomposition structures in a universe of possible metric spaces and possible partitioning functions. Many parallel data problems can't be made to fit into the simple space decomposition structures computer scientists are familiar with.

When massively parallel machinery is the foundational assumption, every computational structure must be represented in terms of space decomposition primitives. Right now, most computer scientists treat space decomposition structures as dumb data buckets and are largely oblivious to their expressiveness. Most published parallel algorithm computer science is based on computational structures that will only sort of parallelize on sufficiently small systems.

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#16

Earlier quoted context omitted.

I understand you may be under NDA, as well as the general pressure to not reveal knowledge that is the basis of your career, but would you care to point out any open literature you think is worthwhile to study?

For massively distributed and parallel algorithms, a core part of the computer science is based around space decomposition data structures. If your algorithm is not based on a space decomposition structure, it won't linearly parallelize across a vast number of cores. When I say some algorithm won't distribute or parallelize, this is what I am usually (but not always) looking at. Space decomposition structures take a…

Are you referring to space-filling curves?

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#17

Earlier quoted context omitted.

For massively distributed and parallel algorithms, a core part of the computer science is based around space decomposition data structures. If your algorithm is not based on a space decomposition structure, it won't linearly parallelize across a vast number of cores. When I say some algorithm won't distribute or parallelize, this is what I am usually (but not always) looking at. Space decomposition structures take a…

Are you referring to space-filling curves?

Not really, though there is a natural relationship between partitioning functions and space-filling curves.

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#18

This looks like a very nice implementation of a graph database. However a 6-machine cluster barely qualifies as "distributed" for the purposes of a graph database. You will experience almost no sub-linearity at this scale no matter how poorly the algorithms distribute. I am not intimately familiar with the transactional implementation but the graph processing algorithms described in Titan are not significantly distri…

I think you are looking at a very different use case here. The systems that I think you are referring to analyze a static graph representation. The Graph500 benchmark in particular loads one big static, unlabeled, undirected, property-free graph and then runs extensive (BFS) analysis algorithms on it. The fact that the graph is not changing allows significant investment into building locality optimizing data structures (which is essentially what space decomposition is all about).

Titan on the other hand is a transactional database system to handle large, multi-relational (labeled) graphs with heterogeneous properties. A Titan graph is constantly evolving (as in the posted benchmark). For graphs (unlike geo-spatial domains), applying space decomposition techniques first requires a metric space embedding which is a non-trivial and computationally expensive process. For changing graphs, this embedding will change as well making this very difficult to use in practice. The best approaches I know of for achieving locality therefore use adaptive graph partitioning techniques instead. However, for the types of OLTP workloads that Titan is optimized for, this would be overkill in the sense that the time spend on partitioning will likely exceed the time saved at runtime. At very large scale, it is most important for OLTP systems to focus on access path optimization based on the ACTUAL query load experienced by the system and not some perceived sense of locality based on connectedness. I published a paper a while ago suggesting one approach to do so: http://www.knowledgefrominformation.com/2010/08/01/cosi-clou... The Graph500 benchmark explicitly prohibits this optimization ("The first kernel constructs an undirected graph in a format usable by all subsequent kernels. No subsequent modifications are permitted to benefit specific kernels").

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#19

This looks like a very nice implementation of a graph database. However a 6-machine cluster barely qualifies as "distributed" for the purposes of a graph database. You will experience almost no sub-linearity at this scale no matter how poorly the algorithms distribute. I am not intimately familiar with the transactional implementation but the graph processing algorithms described in Titan are not significantly distri…

You were using octrees, right? Cat's out of the bag.

Re: The Distributed Graph Database Titan Provides Real-Time Big Graph Data

#20

Earlier quoted context omitted.

I understand you may be under NDA, as well as the general pressure to not reveal knowledge that is the basis of your career, but would you care to point out any open literature you think is worthwhile to study?

For massively distributed and parallel algorithms, a core part of the computer science is based around space decomposition data structures. If your algorithm is not based on a space decomposition structure, it won't linearly parallelize across a vast number of cores. When I say some algorithm won't distribute or parallelize, this is what I am usually (but not always) looking at. Space decomposition structures take a…

> "Right now, most computer scientists treat space decomposition structures as dumb data buckets and are largely oblivious to their expressiveness."

This is very interesting. Is there an example of the expressiveness of spatial decomposition you could share?

Post reply on HN