Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bullshit graph database performance benchmarks

maxdemarzi.com

Post reply on HN
Print view View original

61–70 of 118 posts

Re: Bullshit graph database performance benchmarks

#61

Post by mbuda »

szarnyasg wrote: post #10

A plug: if you are looking for TPC-style application-level benchmarks for database systems, check out the LDBC Social Network Benchmark [1]. It has workloads for both OLTP and OLAP systems. We designed both of these to prevent many of the common benchmarking mistakes. To ensure that implementations follow the specification and their results are reproducible, we have a rigorous auditing process (similarly to TPC's ben…

LDBC is really great, it is and it should be the standard non-biased benchmark to look at. The point with https://memgraph.com/benchgraph is to be more tilted towards some specific workloads, easy to extend and run, etc. Time will tell to which extend that could be achieved, but definitely a place to look for some interesting workloads and results. + the whole thing is super early and it's going to evolve!

Again, definitely take a look at LDBC benchmarks they are in general the most relevant.

Re: Bullshit graph database performance benchmarks

#62

Post by mhio »

robertlagrant wrote: post #37
junon wrote: post #21

Earlier quoted context omitted.

Yeah that was strange, it's my understanding that you can't compare benchmarks between different machines, especially if they're not 1:1 identical hardware. If you're referring to this line, then it struct me as very odd. > Instead of 112 queries per second, I get 531q/s. Instead of a p99 latency of 94.49ms, I get 28ms with a min, mean, p50, p75 and p95 of 14ms to 18ms. Alright, what about query 2? Same story. Otherw…

Author is just stating the differences between the benchmarketing hardware and his own. Not comparing new hardware and one DB with old hardware and other DB.

The bottom table contains the memgraph mgBench (G6 2x Xeon X5650) "hot run, medium, isolated" throughput results:

https://memgraph.com/benchgraph/base?condition=hot&datasetSi...

The "By" columns compares those results to the new test suite on the ~10y newer cpu.

Re: Bullshit graph database performance benchmarks

#63

Post by mbuda »

PreInternet01 wrote: post #3

While this seems to be a pretty egregious example of a vendor benchmark misleading through cherry-picked unrealistic results, I'm not sure I share the author's pessimism about how these kinds of stunts will hold back the graph database market. Why? Simple: pretty much any benchmark I've seen of anything, ever, was similar nonsense -- give people numbers to game and they'll do so, enthusiastically. Even supposedly gol…

"egregious example" is unfair because there was and will be a huge effort in comparing different options. Ofc it's biased. Every single benchmark is biased towards something, but take a look at the specifications and what has actually being compared.

100% agree that everyone should run their own benchmark, that's not possible to do from the vendor perspective, it's just not possible for every single usage. Public benchmarks are something to look into if you want to get cheap info on how different vendors might do for you. Huge but is that it's just a guide, not the only piece of info you should base your decision on.

Re: Bullshit graph database performance benchmarks

#64

Post by mhio »

junon wrote: post #21
mhio wrote: post #13

I get that this is trying to point out that neo4j shouldn't be that far behind, but why are the i7/gatling test numbers being directly compared to memgraphs g6 test results? The conclusion is a bit premature without the other half of the test... What performance does memgraph have on the newer, single socket hardware?

Yeah that was strange, it's my understanding that you can't compare benchmarks between different machines, especially if they're not 1:1 identical hardware. If you're referring to this line, then it struct me as very odd. > Instead of 112 queries per second, I get 531q/s. Instead of a p99 latency of 94.49ms, I get 28ms with a min, mean, p50, p75 and p95 of 14ms to 18ms. Alright, what about query 2? Same story. Otherw…

It was...

> It looks like Neo4j is faster than Memgraph in the Aggregate queries by about 3 times. Memgraph is faster than Neo4j for the queries they selected by about 2-3x except...

Unless that's meant to be a joke? Maybe they were dunking on the "bullshit" benchmark with a worse comparison.

Re: Bullshit graph database performance benchmarks

#65

Post by junon »

robertlagrant wrote: post #37
junon wrote: post #21

Earlier quoted context omitted.

Yeah that was strange, it's my understanding that you can't compare benchmarks between different machines, especially if they're not 1:1 identical hardware. If you're referring to this line, then it struct me as very odd. > Instead of 112 queries per second, I get 531q/s. Instead of a p99 latency of 94.49ms, I get 28ms with a min, mean, p50, p75 and p95 of 14ms to 18ms. Alright, what about query 2? Same story. Otherw…

Author is just stating the differences between the benchmarketing hardware and his own. Not comparing new hardware and one DB with old hardware and other DB.

But those are needless comparisons that make no sense to even mention. Of course the performance profile is different. That was the original point, if I understand correctly.

EDIT: GP followed up, I did not in fact understand correctly.

Re: Bullshit graph database performance benchmarks

#66

Post by junon »

mhio wrote: post #64
junon wrote: post #21

Earlier quoted context omitted.

Yeah that was strange, it's my understanding that you can't compare benchmarks between different machines, especially if they're not 1:1 identical hardware. If you're referring to this line, then it struct me as very odd. > Instead of 112 queries per second, I get 531q/s. Instead of a p99 latency of 94.49ms, I get 28ms with a min, mean, p50, p75 and p95 of 14ms to 18ms. Alright, what about query 2? Same story. Otherw…

It was... > It looks like Neo4j is faster than Memgraph in the Aggregate queries by about 3 times. Memgraph is faster than Neo4j for the queries they selected by about 2-3x except... Unless that's meant to be a joke? Maybe they were dunking on the "bullshit" benchmark with a worse comparison.

Oh, then I'm not sure what you mean. That line makes sense - the final benchmarks were performed on the author's machine. That's where the conclusion comes from.

In theory, the spread between two benchmarked programs is not going to be hugely different between machines unless one is taking advantage of the hardware of one of the machines where the other doesn't (e.g. new syscall mechanisms such as io_uring, SIMD support, multithreading in some cases, etc).

2-3x is a much more reasonable spread than 100x. If they really did have 100x speedups, then the culprit may be the fact they're using obscenely old hardware, which would be disingenuous given that people are not typically running graph databases on such infrastructure anymore.

Re: Bullshit graph database performance benchmarks

#67

Post by rjh29 »

steve_gh wrote: post #56
rjh29 wrote: post #34

Earlier quoted context omitted.

For simple cases, you can get pretty far storing relations 6 times in SQLite, or any old key/value store. (a-b-son, b-a-father, son-a-b, father-b-a, a-son-b, b-father-a)

This is interesting - can you expand a little or provide a link. I get a-b links, but where do father and son come in?

I think I read it in a tech company's blog post, but there are some Wikipedia articles on the subject:

https://en.wikipedia.org/wiki/Triplestore

https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...

The father/son is the relation between the two nodes, to support multiple relation types.

Re: Bullshit graph database performance benchmarks

#68

Post by ramraj07 »

alias_neo wrote: post #27
dwroberts wrote: post #23

The author works on RageDB ( https://ragedb.com/ ) and this doesn't seem to be disclosed in the article

I'm not sure why it would need to be "disclosed", other than to suggest the author knows what they're talking about due to "domain knowledge".

Maybe because I don’t trust someone who allegedly writes databases but is proud about not knowing python.

Re: Bullshit graph database performance benchmarks

#69

Post by apavlo »

college_physics wrote: post #51

my feeling is that graph databases face an uphill battle for mass adoption not because their architects or vendors doing anything wrong but some intrinsic aspects of information exchange in most current situations and use cases * information tends to be private and/or commercially sensitive, this severs the links that graph dbs are good at representing (and made the "node focused" SQL approach the ubiquitous model th…

> but maybe what results is a relational database+ type thingy

This is called already called "object-relational" model. It was invented by Postgres in the 1980s. The relational model / SQL absorbs the best part of alternative systems and get better over time. SQL:2023 is adding support for graph queries (SQL/PCG).

Graph DBMSs are a passing fade.

Re: Bullshit graph database performance benchmarks

#70

Post by bjornsing »

> Why would they do this? Because it’s a bullshit benchmark and they don’t actually want anybody looking too deeply at it.

Very unlikely I would say. Most likely ”the clowns” never considered how the license terms would impact use of benchmarks.

“Never attribute to malice why can be sufficiently explained by incompetence” as the saying goes.

Post reply on HN