Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres as a Graph Database: (Ab)Using PgRouting

supabase.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#21

Post by wslh »

ashvardanian wrote: post #7

Five years ago I was absolutely frustrated with the state of Graph databases and libraries and tried putting several non-Graph DBMSs behind a NetworkX-like Python interface https://github.com/unum-cloud/NetworkXum >. When benchmarked, Neo4J crashed on every graph I’ve tried https://www.unum.cloud/blog/2020-11-12-graphs >, making SQLite and Postgres much more viable options even for network-processing workloads. So I…

Have you tried NetworkDisk[1] to manipulate NetworkX graphs on disk?

[1] https://networkdisk.inria.fr/

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#22

Post by threeseed »

ashvardanian wrote: post #7

Five years ago I was absolutely frustrated with the state of Graph databases and libraries and tried putting several non-Graph DBMSs behind a NetworkX-like Python interface https://github.com/unum-cloud/NetworkXum >. When benchmarked, Neo4J crashed on every graph I’ve tried https://www.unum.cloud/blog/2020-11-12-graphs >, making SQLite and Postgres much more viable options even for network-processing workloads. So I…

You ran Neo4J with 512MB even thought it has always recommended 2GB at a minimum.

And MemGraph is nice but it's memory only where as Neo4J is designed for super large graphs that live on the filesystem. Not really that comparable.

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#24

Post by threeseed »

nemo44x wrote: post #12
ashvardanian wrote: post #7

Five years ago I was absolutely frustrated with the state of Graph databases and libraries and tried putting several non-Graph DBMSs behind a NetworkX-like Python interface https://github.com/unum-cloud/NetworkXum >. When benchmarked, Neo4J crashed on every graph I’ve tried https://www.unum.cloud/blog/2020-11-12-graphs >, making SQLite and Postgres much more viable options even for network-processing workloads. So I…

Neo4j is pretty bad and very dated. No idea what they’re doing. MemGraph is a much better tool. Really graph is a feature and not a product.

Neo4J is mature not dated which is why it's so popular.

And couldn't disagree more that graph is a feature. You really want something optimised for it (query language / storage approach) as the data structure is so different in every way from a relational or document store.

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#25

Post by lmeyerov »

whalesalad wrote: post #5

Has anyone done something like this with a very large dataset? Hundreds of millions of rows.

For sub-billion graph queries, we have been doing in-memory via GFQL that runs graph queries & analytics on top of pandas (columnar CPU dataframes) and cudf (columnar GPU dataframes). Sub-billion is typically small enough that single node is fine. We began dev after years of being annoyed at having no good simple OSS in-process solution for small graphs like these. Avoiding jumping architectural hoops around dealing with multiple systems of record infra is nice, as can often just do in the compute tier.

Once big graphs get involved, scalable systems, and especially those that seperate storage from compute & price accordingly, get much more interesting. We work with our partners like databricks, Google spanner, AWS Neptune, etc, who have different sweet spots that really depend on workload and context, they're all pretty different. OLTP vs OLAP, etc.

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#26

Post by tucnak »

threeseed wrote: post #24
nemo44x wrote: post #12

Earlier quoted context omitted.

Neo4j is pretty bad and very dated. No idea what they’re doing. MemGraph is a much better tool. Really graph is a feature and not a product.

Neo4J is mature not dated which is why it's so popular. And couldn't disagree more that graph is a feature. You really want something optimised for it (query language / storage approach) as the data structure is so different in every way from a relational or document store.

How is it different? Isn't a graph basically two sets of tuples: edges and nodes? I played with Cayley (Google) for a little while, & that was my impression.

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#27

Post by gkorland »

ashvardanian wrote: post #7

Five years ago I was absolutely frustrated with the state of Graph databases and libraries and tried putting several non-Graph DBMSs behind a NetworkX-like Python interface https://github.com/unum-cloud/NetworkXum >. When benchmarked, Neo4J crashed on every graph I’ve tried https://www.unum.cloud/blog/2020-11-12-graphs >, making SQLite and Postgres much more viable options even for network-processing workloads. So I…

Your graph DB frustrations mirror what many experienced with Neo4j. If you refresh your project, consider including FalkorDB (formerly RedisGraph) - it uses sparse adjacency matrices and GraphBLAS for much better performance while supporting Cypher.

Would be interesting to see updated benchmarks comparing these newer options against PostgreSQL extensions.

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#29

Post by philjohn »

ashvardanian wrote: post #7

Five years ago I was absolutely frustrated with the state of Graph databases and libraries and tried putting several non-Graph DBMSs behind a NetworkX-like Python interface https://github.com/unum-cloud/NetworkXum >. When benchmarked, Neo4J crashed on every graph I’ve tried https://www.unum.cloud/blog/2020-11-12-graphs >, making SQLite and Postgres much more viable options even for network-processing workloads. So I…

A good 10 years ago or so I was running a solution that used RDF Quad Stores - and the best one at the time (after trialling 4Store, Marklogic and some others I can't remember) was OpenLink Virtuoso - how they managed to fit a performant distributed Quad store into what started life as an SQL engine was impressive.

I've left that world now, but if you're in the market for a graph store again, it might be something to look at.

Re: Postgres as a Graph Database: (Ab)Using PgRouting

#30

Post by bubblyworld »

tucnak wrote: post #26
threeseed wrote: post #24

Earlier quoted context omitted.

Neo4J is mature not dated which is why it's so popular. And couldn't disagree more that graph is a feature. You really want something optimised for it (query language / storage approach) as the data structure is so different in every way from a relational or document store.

How is it different? Isn't a graph basically two sets of tuples: edges and nodes? I played with Cayley (Google) for a little while, & that was my impression.

I think it's less a matter of "can you represent graphs in a relational DB" (of course you can), and more about what kind of queries the DB is optimised for. Graph databases are intended for complex recursive queries on relatively unstructured data. You could certainly do that in SQL if you wanted to, but you'll pay for it performance-wise.

Graph query languages also make those kinds of queries much easier to express in the first place.

Post reply on HN