Unicorn: a simple and flexible abstraction of BigTable-like databases • Hacker News Board

Re: Unicorn: a simple and flexible abstraction of BigTable-like databases

#21

Post by janprill » Mon, May 23, 2016, 8:37 PM UTC

janprill wrote: post #20

While people seem to mostly tinker with the name in the comments I'd like to say that this looks like a really interesting project! Would you mind to give us a little more background with regards to how this has been initiated, what your motivation was to write something new? Given that you have an interesting vita ( https://www.linkedin.com/in/haifengli ) and a lot of people are interested in the graph database spac…

Ok, doing a little research about ADP I realize that this is quite a large company. Sorry, I didn't knew it (am from Germany). But this would make it even more interesting how unicorn is used at ADP and if this already is a reference with regards to the scale of larger installations of Unicorn.

Re: Unicorn: a simple and flexible abstraction of BigTable-like databases

#22

Post by lvca » Tue, May 24, 2016, 11:25 AM UTC

rspeer wrote: post #2

How would I use this if I have graph data that's described in terms of its edges, not its nodes? The N-Triples and DOT formats would be examples of graph data that's structured like this: you just list the edges as the pairs of nodes that they connect. The nodes don't necessarily have any properties, they're just implicitly created by edges. I could describe a -- b b -- c b -- d and nodes "a", "b", "c", and "d" would…

About the usage of ETL and OrientDB, what's your input file? I'd like to understand current limitation of OrientDB ETL.

Re: Unicorn: a simple and flexible abstraction of BigTable-like databases

#23

Post by rspeer » Tue, May 24, 2016, 4:53 PM UTC

lvca wrote: post #22

rspeer wrote: post #2

How would I use this if I have graph data that's described in terms of its edges, not its nodes? The N-Triples and DOT formats would be examples of graph data that's structured like this: you just list the edges as the pairs of nodes that they connect. The nodes don't necessarily have any properties, they're just implicitly created by edges. I could describe a -- b b -- c b -- d and nodes "a", "b", "c", and "d" would…

About the usage of ETL and OrientDB, what's your input file? I'd like to understand current limitation of OrientDB ETL.

It's in this bug report: https://github.com/orientechnologies/orientdb/issues/6179

I recognize this was closed (by you, apparently) on the basis that I was using a bug report to ask a question. The intent of the bug report is that if the ETL file doesn't sufficiently describe what to do with the data, you should display a sensible error message, instead of a traceback from deep within the internals of OrientDB. If I've failed to list a primary key somewhere, you should tell me that, not assume that I want the primary key to be "null".

But yes, on top of that, my question remains.

Re: Unicorn: a simple and flexible abstraction of BigTable-like databases

#24

Post by haifeng » Fri, May 27, 2016, 2:23 PM UTC

rspeer wrote: post #17

haifeng wrote: post #4

Earlier quoted context omitted.

In most graph database, you find a vertex by filtering its properties, e.g. Gremlin graph query language. In Unicorn, you can do the similar with document vertices (it is, a vertex corresponding to a document in another table/collection). This is probably very nature in a business application. However, it is not very useful in your case as your vertices are abstract without any properties. I guess what you want is so…

I would say that what I have is a property-directed multi-graph, as I understand it. It's just that the properties are on the edges, and the nodes have no properties except for their ID. The graph in question is ConceptNet, which in the version I'm working on has about 10 million edges and 3 million nodes. Let's be clear that, in computing, "million" is not a large number. I only said "large graph" to clarify that it…

Check out this script https://github.com/haifengl/unicorn/blob/master/shell/src/un..., which loads dbpedia graph into unicorn. You should be able to load ConceptNet without minor modifications. Later, you can refer a vertex by its string id.