Previously discussed at https://news.ycombinator.com/item?id=7946024.
Cayley – An open-source graph database
41–50 of 60 posts
Re: Cayley – An open-source graph database
#42Earlier quoted context omitted.
https://github.com/neo4j/neo4j
Neo4j, like all graph databases I've tried, is only okay with small data. Suppose I want to import a medium-sized graph into Neo4j. Medium-sized as in "fits on a hard disk and doesn't quite fit in RAM". One example would be importing DBPedia. Some people have come up with not-very-supported hacks for loading DBPedia into Neo4j. Some StackOverflow comments such as [1] will point you to them, and the GitHub pages will…
>>And the last time I tried to load my multi-million-edge dataset into Neo4j through its documented API, I estimated that it would have taken several weeks to finish.
Use the Import tool, it can do a million writes a second. Here is how to import hacker news into Neo4j using it: https://maxdemarzi.com/2015/04/14/importing-the-hacker-news-...
Re: Cayley – An open-source graph database
#43What exactly is an open source graph?
Re: Cayley – An open-source graph database
#44Earlier quoted context omitted.
Neo4j, like all graph databases I've tried, is only okay with small data. Suppose I want to import a medium-sized graph into Neo4j. Medium-sized as in "fits on a hard disk and doesn't quite fit in RAM". One example would be importing DBPedia. Some people have come up with not-very-supported hacks for loading DBPedia into Neo4j. Some StackOverflow comments such as [1] will point you to them, and the GitHub pages will…
Very interesting. I think the reality in a lot of situations is that most people don't really need the full feature-set that graph databases provide. I ran into a similar problem trying to explore Wikidata's json dumps. It's a lot simpler to load it into MongoDB and create indices where you need them, rather than figuring out how to interface to a proprietary system that you may or may not end up using in the long ru…
What I end up with is a 4 GB index, whose contents are byte offsets into an 8 GB file containing the properties of the edges.
When I mmap these, an in-memory lookup takes about 1 ms, but I can have unfortunate delays of like 100 ms per lookup if I'm starting cold or hitting things that are swapped out of memory.
Re: Cayley – An open-source graph database
#45Earlier quoted context omitted.
Neo4j, like all graph databases I've tried, is only okay with small data. Suppose I want to import a medium-sized graph into Neo4j. Medium-sized as in "fits on a hard disk and doesn't quite fit in RAM". One example would be importing DBPedia. Some people have come up with not-very-supported hacks for loading DBPedia into Neo4j. Some StackOverflow comments such as [1] will point you to them, and the GitHub pages will…
Very interesting. I think the reality in a lot of situations is that most people don't really need the full feature-set that graph databases provide. I ran into a similar problem trying to explore Wikidata's json dumps. It's a lot simpler to load it into MongoDB and create indices where you need them, rather than figuring out how to interface to a proprietary system that you may or may not end up using in the long ru…
Also, yeah, I don't really need most of the things that graph DBs are offering. They seem to focus a lot on making computationally unreasonable things possible -- such as arbitrary SPARQL queries.
I'm not the kind of madman who wants to do arbitrary SPARQL queries. I just want to be able to load data quickly, look up the edges around a node quickly, and then once I can do that, I'd also like to be able to run the occasional algorithm that propagates across the entire graph, like finding the degree-k core.
Re: Cayley – An open-source graph database
#46Earlier quoted context omitted.
Neo4j, like all graph databases I've tried, is only okay with small data. Suppose I want to import a medium-sized graph into Neo4j. Medium-sized as in "fits on a hard disk and doesn't quite fit in RAM". One example would be importing DBPedia. Some people have come up with not-very-supported hacks for loading DBPedia into Neo4j. Some StackOverflow comments such as [1] will point you to them, and the GitHub pages will…
>>And the last time I tried to load my multi-million-edge dataset into Neo4j through its documented API, I estimated that it would have taken several weeks to finish. Use the Import tool, it can do a million writes a second. Here is how to import hacker news into Neo4j using it: https://maxdemarzi.com/2015/04/14/importing-the-hacker-news-...
Thanks, I'll put that on my list of things to try, although I've spent more than enough time banging my head against graph DBs for today.
This isn't insurmountable, but I'm just going to gripe about it: I'm annoyed by the idea that I need to make a table of nodes and load it in first. Every graph DB tutorial seems to do this, because it looks like what you'd do if you were moving your relational data into a graph DB. But I have RDF-like data where nodes are just inconsequential string IDs.
Hm, this indicates that I should definitely be looking at Cayley, which directly supports loading RDF quads.
Re: Cayley – An open-source graph database
#47The project has no activity in the last 6 months. I would like to see an open-source graph database that's being actively developed and maintained.
Blazegraph. As used for wikidata among others. Has lots of property graph support not just semantic graphs.
While I've been pointing out the problems I've encountered with Neo4J and OrientDB, I should say as a counterpoint that I just tried Blazegraph based on this recommendation, and so far, it works.
Its importer was not the most intuitive thing to use -- I had to dig up configuration items that were scattered across its documentation and Stack Overflow -- but I got it to work, it imported 25 million edges in less than 8 minutes, and it's providing reasonably quick access to those edges now.
Re: Cayley – An open-source graph database
#48Earlier quoted context omitted.
Neo4j, like all graph databases I've tried, is only okay with small data. Suppose I want to import a medium-sized graph into Neo4j. Medium-sized as in "fits on a hard disk and doesn't quite fit in RAM". One example would be importing DBPedia. Some people have come up with not-very-supported hacks for loading DBPedia into Neo4j. Some StackOverflow comments such as [1] will point you to them, and the GitHub pages will…
Very interesting. I think the reality in a lot of situations is that most people don't really need the full feature-set that graph databases provide. I ran into a similar problem trying to explore Wikidata's json dumps. It's a lot simpler to load it into MongoDB and create indices where you need them, rather than figuring out how to interface to a proprietary system that you may or may not end up using in the long ru…
I use MongoDB for a Wikidata replica and index performances are quite good. I use some hacks in order to keep size of indexed values low (see https://github.com/ProjetPP/WikibaseEntityStore/blob/master/... ). It helps a lot in order to be able to keep indexes into memory.
It powers https://askplatyp.us quite well.
Re: Cayley – An open-source graph database
#49The project has no activity in the last 6 months. I would like to see an open-source graph database that's being actively developed and maintained.
https://github.com/amark/gun
Open Source, graph database, P2P/decentralized, 1800+ stars, browser/javascript friendly, active community, does realtime updates like Firebase.
With some INSANE performance specs:
https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...
Re: Cayley – An open-source graph database
#50For people that are concerned about lack of development on Cayley it looks like there is a fork that is actively merging PRs if nothing else:
https://github.com/dennwc/cayley
Found from the mailing list:
https://groups.google.com/forum/?hl=en#!topic/cayley-users/D...