Bullshit graph database performance benchmarks
maxdemarzi.com
Bullshit graph database performance benchmarks
1–10 of 118 posts
Re: Bullshit graph database performance benchmarks
#2Thanks for digging and sharing, I enjoyed your snark.
> They decided to provide the data not in a CSV file like a normal human being would, but instead in a giant cypher file performing individual transactions for each node and each relationship created. Not batches of transactions… but rather painful, individual, one at a time transactions one point 8 million times. So instead of the import taking 2 minutes, it takes hours.
Yeahhh I noticed this too when I looked at the repo when their blog was posted a couple weeks back. Running a transaction for each object will of course be very slow and real production code will (hopefully) not do this.
> Those are not “graphy” queries at all, why are they in a graph database benchmark? Ok, whatever.
I’m definitely interested in seeing more realistic scenarios of actual “graphy” queries with batched transactions comparing the two. Oh, and comparing against Neptune would be cool too since that supposedly uses openCypher now (which I hear is kinda close to neo4j cypher?).
Re: Bullshit graph database performance benchmarks
#3While this seems to be a pretty egregious example of a vendor benchmark misleading through cherry-picked unrealistic results, I'm not sure I share the author's pessimism about how these kinds of stunts will hold back the graph database market.
Why? Simple: pretty much any benchmark I've seen of anything, ever, was similar nonsense -- give people numbers to game and they'll do so, enthusiastically. Even supposedly gold-standard benchmarks like the TechEmpower framework benchmarks quickly devolve into "application server handling HTTP requests by responding with predefined strings", which is as fast as it's utterly useless in most people's version of the real world.
The only way to get usable benchmark data is to run your own workloads in your own environment: everything else is pretty much noise.
Re: Bullshit graph database performance benchmarks
#4For more context:
- blog post that sparked the discussion - https://memgraph.com/blog/memgraph-vs-neo4j-performance-benc...
- earlier discussion about this Memgraph benchmark HackerNews - https://news.ycombinator.com/item?id=33813781
- the benchmark results - https://memgraph.com/benchgraph/
- benchmark repo and methodology - https://github.com/memgraph/memgraph/tree/master/tests/mgben...
Re: Bullshit graph database performance benchmarks
#5For more context: - blog post that sparked the discussion - https://memgraph.com/blog/memgraph-vs-neo4j-performance-benc... - earlier discussion about this Memgraph benchmark HackerNews - https://news.ycombinator.com/item?id=33813781 - the benchmark results - https://memgraph.com/benchgraph/ - benchmark repo and methodology - https://github.com/memgraph/memgraph/tree/master/tests/mgben...
[deleted]
Re: Bullshit graph database performance benchmarks
#6Ah yes, memgraph.
I argued with them here about another bullshit benchmark https://news.ycombinator.com/item?id=33717766
they did reply tho
Re: Bullshit graph database performance benchmarks
#7While this seems to be a pretty egregious example of a vendor benchmark misleading through cherry-picked unrealistic results, I'm not sure I share the author's pessimism about how these kinds of stunts will hold back the graph database market. Why? Simple: pretty much any benchmark I've seen of anything, ever, was similar nonsense -- give people numbers to game and they'll do so, enthusiastically. Even supposedly gol…
Yes, running a benchmark on your data is the only way. I've taken a look at both benchmarks (the one from OP and the one from Memgraph). They seem like different types of benchmarks and different approaches. But I still find it interesting that although the numbers in OP's are not so much in favor of Memgraph it turns out that Memgrpah is faster than Neo4j in large number of benchmark queries. So yes, it all comes down to type of benchmark and data that you use.
I've also noticed (from OPs tweet https://twitter.com/maxdemarzi/status/1613075177704677376) that he used Enterprise version of Neo4j, but it doesn't say which Memgrpah version was used. I don't have experience with this two databases, but usually ENT versions are somewhat better than community ones.
[EDIT]: I fixed few typos.
Re: Bullshit graph database performance benchmarks
#8On a tangent, what Graph Database would people recommend in 2023? In particular, I would like something that's linked in like SQLite rather than a full blown service like MySQL etc
Re: Bullshit graph database performance benchmarks
#9On a tangent, what Graph Database would people recommend in 2023? In particular, I would like something that's linked in like SQLite rather than a full blown service like MySQL etc
Kuzu looks very interesting: https://github.com/kuzudb/kuzu
Discussed here yesterday: https://news.ycombinator.com/item?id=34358912
Re: Bullshit graph database performance benchmarks
#10A plug: if you are looking for TPC-style application-level benchmarks for database systems, check out the LDBC Social Network Benchmark [1]. It has workloads for both OLTP and OLAP systems. We designed both of these to prevent many of the common benchmarking mistakes. To ensure that implementations follow the specification and their results are reproducible, we have a rigorous auditing process (similarly to TPC's benchmarks) [2].
[1] https://ldbcouncil.org/docs/presentations/ldbc-snb-2022-11.p...