Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bullshit graph database performance benchmarks

maxdemarzi.com

Post reply on HN
Print view View original

51–60 of 118 posts

Re: Bullshit graph database performance benchmarks

#51

Post by college_physics »

my feeling is that graph databases face an uphill battle for mass adoption not because their architects or vendors doing anything wrong but some intrinsic aspects of information exchange in most current situations and use cases

* information tends to be private and/or commercially sensitive, this severs the links that graph dbs are good at representing (and made the "node focused" SQL approach the ubiquitous model that it currently is)

* objects in typical schemas have many more attributes that relations. while you could model things RDF style where everything is a relation, it is not the most intuitive for people

* when the previous constraint does not apply (e.g. data from a centralized social network), it is typically not too hard to emulate an adequate graph structure on a Pareto 20/80 basis using an RDBMS

so graph dbs end up being optimal only for a niche of situations and probably not the impact that the people / investors involved in their development would be happy with

on the other hand the ginie is out of the bottle and the decades-long SQL monoculture seems to be coming to an end. but maybe what results is a relational database+ type thingy [0] rather than two disconnected paradigms

[0] https://postgresconf.org/conferences/2020/program/proposals/...

Re: Bullshit graph database performance benchmarks

#52

Post by szarnyasg »

ashvardanian wrote: post #46

I am really stunned by this story. It made me check the MemGraph benchmarks section. Don't get me wrong, it may be 10-100x faster than Neo4J in even the most basic operations. Moreover, given the quality of Neo4J, it is hard not to be that much quicker. Even Postgres and MySQL are better at storing graphs than Neo4J. --- Disclosure: I have worked on Graph Algorithms, Graph Databases, and Database Engines for years, a…

In LDBC, we have a separate benchmark for graph algorithms: https://ldbcouncil.org/benchmarks/graphalytics/

This focuses on untyped, unattributed graphs and includes algorithms which are often formulated following a vertex-centric programming model (BFS, PageRank, community detection, connected components, etc.). The SNB Business Intelligence already covers a few of these algorithms (BFS, weighted shortest paths) and in the future it may incorporate more.

We plan to run a Graphalytics competition in the spring (on data sets with up to tens of billions of edges) - let me know if anyone is interested in participating in this.

Re: Bullshit graph database performance benchmarks

#53

Post by ashvardanian »

szarnyasg wrote: post #52
ashvardanian wrote: post #46

I am really stunned by this story. It made me check the MemGraph benchmarks section. Don't get me wrong, it may be 10-100x faster than Neo4J in even the most basic operations. Moreover, given the quality of Neo4J, it is hard not to be that much quicker. Even Postgres and MySQL are better at storing graphs than Neo4J. --- Disclosure: I have worked on Graph Algorithms, Graph Databases, and Database Engines for years, a…

In LDBC, we have a separate benchmark for graph algorithms: https://ldbcouncil.org/benchmarks/graphalytics/ This focuses on untyped, unattributed graphs and includes algorithms which are often formulated following a vertex-centric programming model (BFS, PageRank, community detection, connected components, etc.). The SNB Business Intelligence already covers a few of these algorithms (BFS, weighted shortest paths) and…

Sounds good, we may participate. Will you be adding larger graph datasets to the list?

Re: Bullshit graph database performance benchmarks

#54

Post by szarnyasg »

ashvardanian wrote: post #53
szarnyasg wrote: post #52

Earlier quoted context omitted.

In LDBC, we have a separate benchmark for graph algorithms: https://ldbcouncil.org/benchmarks/graphalytics/ This focuses on untyped, unattributed graphs and includes algorithms which are often formulated following a vertex-centric programming model (BFS, PageRank, community detection, connected components, etc.). The SNB Business Intelligence already covers a few of these algorithms (BFS, weighted shortest paths) and…

Sounds good, we may participate. Will you be adding larger graph datasets to the list?

Not this time, the largest data sets are going to be graph500-30 and datagen-sf10k-fb. They have 17B and 9.4B undirected edges, respectively.

Re: Bullshit graph database performance benchmarks

#55

Post by oxfordmale »

alexott wrote: post #50
oxfordmale wrote: post #43

Benchmarks are generally useless unless they test real world scenarios. The DataBricks data warehouse record costed $5,190,345 USD to run over a period of 3 years. If I spend that amount of money, I will get fired. Such benchmarks also ignore the engineering expertise an organisation has. Do you need to be an expert to fine tune 6000 parameters or can you tune the system to an acceptable standard by reading a few blo…

That number should be "The total 3-year price of the entire Priced Configuration must be reported, including: hardware, software, and maintenance charges", so they just took the cost of the hardware used for benchmark, and extended it to 3 years. If you look into the blog post: https://www.databricks.com/blog/2021/11/02/databricks-sets-o... , you will see that it costed $242

Yes the final run, establishing the record, costed $242. I would love to know what the total compute costs for this project was. In real world situations, you run this query daily, or at least multiple times to fine tune it. The point still stands that I can't afford to run on this type of hardware, as it is too expensive, nor do I have such heavy workloads, so these results are not relevant.

Re: Bullshit graph database performance benchmarks

#56

Post by steve_gh »

rjh29 wrote: post #34
alfiedotwtf wrote: post #8

On a tangent, what Graph Database would people recommend in 2023? In particular, I would like something that's linked in like SQLite rather than a full blown service like MySQL etc

For simple cases, you can get pretty far storing relations 6 times in SQLite, or any old key/value store. (a-b-son, b-a-father, son-a-b, father-b-a, a-son-b, b-father-a)

This is interesting - can you expand a little or provide a link. I get a-b links, but where do father and son come in?

Re: Bullshit graph database performance benchmarks

#57

Post by olavgg »

mcv wrote: post #36
chillfox wrote: post #17

Earlier quoted context omitted.

If you only need a few graph queries then you could just use SQLite, it’s capable of doing it (I have done it before). But writing graph queries in SQL is painful, so I wouldn’t do it if you need more than a handful.

Exactly. There's no need for a graph database if you're not going to be searching for complex relationships. And once you do, SQL is hell. I don't know of any SQLite-like graph database. I'm still a fan of neo4j.

I also really like Neo4J, but the community version is very crippled. Still though, I've worked around it and do hot backups with ZFS and run multiple database processes to support multiple customers.

Re: Bullshit graph database performance benchmarks

#58

Post by oxfordmale »

salt-shaker wrote: post #49
oxfordmale wrote: post #43

Benchmarks are generally useless unless they test real world scenarios. The DataBricks data warehouse record costed $5,190,345 USD to run over a period of 3 years. If I spend that amount of money, I will get fired. Such benchmarks also ignore the engineering expertise an organisation has. Do you need to be an expert to fine tune 6000 parameters or can you tune the system to an acceptable standard by reading a few blo…

I was curious about it, so I tried to figure out where you got this number from. It looks like your source is https://www.tpc.org/results/individual_results/databricks/da... , but you interpreted it wrong. The number you quoted is the projected 3-year ownership of the system configuration that was used to run the test, so the actual cost is a small fraction of the number you quoted.

It is worth noting the compute cost appears to be based on purchasing reserved instances from AWS. The price of on demand instances is much higher.

The laptop is also very low budget. I am sure it is fine to run the final query, however, you would unlikely to be able to use that as a development machine.

Re: Bullshit graph database performance benchmarks

#59

Post by alfiedotwtf »

mcv wrote: post #36
chillfox wrote: post #17

Earlier quoted context omitted.

If you only need a few graph queries then you could just use SQLite, it’s capable of doing it (I have done it before). But writing graph queries in SQL is painful, so I wouldn’t do it if you need more than a handful.

Exactly. There's no need for a graph database if you're not going to be searching for complex relationships. And once you do, SQL is hell. I don't know of any SQLite-like graph database. I'm still a fan of neo4j.

Yeah, Neo4J is neat but I don't want to stand up services... that's why I wondered if there were some that linked in easily

Re: Bullshit graph database performance benchmarks

#60

Post by mbuda »

ashvardanian wrote: post #46

I am really stunned by this story. It made me check the MemGraph benchmarks section. Don't get me wrong, it may be 10-100x faster than Neo4J in even the most basic operations. Moreover, given the quality of Neo4J, it is hard not to be that much quicker. Even Postgres and MySQL are better at storing graphs than Neo4J. --- Disclosure: I have worked on Graph Algorithms, Graph Databases, and Database Engines for years, a…

A. If you take a deeper look, the benchmarking part is implemented in C++ (client + benchmark session management + measurements), Python is just layer on top to orchestrate everything but it's not on a critical path at all

B. Yes, old hardware is 100% a problem, that's why the benchgraph web is extensible with more hardware options, stay tuned for that, it's going to come soon!

C. Yep, legit, something to expand / improve

Post reply on HN