my feeling is that graph databases face an uphill battle for mass adoption not because their architects or vendors doing anything wrong but some intrinsic aspects of information exchange in most current situations and use cases * information tends to be private and/or commercially sensitive, this severs the links that graph dbs are good at representing (and made the "node focused" SQL approach the ubiquitous model th…
> but maybe what results is a relational database+ type thingy This is called already called "object-relational" model. It was invented by Postgres in the 1980s. The relational model / SQL absorbs the best part of alternative systems and get better over time. SQL:2023 is adding support for graph queries (SQL/PCG). Graph DBMSs are a passing fade.
Bullshit graph database performance benchmarks
81–90 of 118 posts
Re: Bullshit graph database performance benchmarks
#82Earlier quoted context omitted.
Oh, then I'm not sure what you mean. That line makes sense - the final benchmarks were performed on the author's machine. That's where the conclusion comes from. In theory, the spread between two benchmarked programs is not going to be hugely different between machines unless one is taking advantage of the hardware of one of the machines where the other doesn't (e.g. new syscall mechanisms such as io_uring, SIMD supp…
> the final benchmarks were performed on the author's machine No, the author didn't run any benchmarks for Memgraph AFAICT, only for Neo4j. The numbers for mempgraph at the end are from the old benchmark, so on the old hardware.
Re: Bullshit graph database performance benchmarks
#83Now, let's combine this with one of the persistent tendencies of developers to take one specific benchmark as indicative of the overall performance, which is often preyed on by benchmarkers trying to sell things.
Is it really plausible that neo4j takes 120x longer than it needs to on all operations? A dedicated graph database that has been tuned and optimized for that task for quite a while now?
I'm not quite going to rate that a 0 probability, but it's definitely a very big claim. While the probability is not 0, it is comfortably below "someone's gaming the numbers" and "the benchmark is not as comparable as claimed". There's a faint chance the latter may match a production use case; for instance, certain comparisons of NoSQL DBs and SQL DBs are "not fair" in that they won't be doing remotely the same things for the queries and the performance landscape is very complicated, with one side winning handily for some tasks and the other side handily winning for others, but if your use case falls into one of those big wins you may not care about the "fairness". But it's still a pretty big chunk of probability mass that it's just plain not comparable; how many times have we seen a ludicrous benchmarking claim of relative superiority just for the losing side to pop up and say something to the effect of "Hey, did you consider adding the correct index to the data, oh look if you do that we win by a factor of 4."
Tell me you're 1.2x or 1.5x faster or something, or that your clever compression means I can remove 1/3rd of my systems or something. Keep it in the range of plausible.
While I'm sure this won't affect the marketing of this company any, ludicrously large claims of 10x+ speed improvements actually turn me off, not attract me. You'd better have some sort of super compelling reason why you somehow managed to be that fast over your competitor, like, "we're the first to successfully leverage GPUs" or something like that. Otherwise I'm going to guess "Actually, you have an O(log n log log n) algorithm over their O(log n log n) algorithm and you cranked the data set up to the ludicrous sizes it takes to get an arbitrarily large X factor improvement over your competition" or something like that.
(Always gotta love people comparing two completely different O(...) algorithms against each other and declaring one is X times faster than the other. This is another major source of "10,000x faster!"... yeah, O(n log n) is "10,000 faster!" than O(n^2), sure. It's also 100,000 times faster, 10 times faster, and a billionkajillion times faster, all at the same time.)
Re: Bullshit graph database performance benchmarks
#84my feeling is that graph databases face an uphill battle for mass adoption not because their architects or vendors doing anything wrong but some intrinsic aspects of information exchange in most current situations and use cases * information tends to be private and/or commercially sensitive, this severs the links that graph dbs are good at representing (and made the "node focused" SQL approach the ubiquitous model th…
> but maybe what results is a relational database+ type thingy This is called already called "object-relational" model. It was invented by Postgres in the 1980s. The relational model / SQL absorbs the best part of alternative systems and get better over time. SQL:2023 is adding support for graph queries (SQL/PCG). Graph DBMSs are a passing fade.
Re: Bullshit graph database performance benchmarks
#85Completely useless tangent: the word "benchmark" comes from a mark that surveyors would make in rock so that they could place a leveling rod for surveying. Benchmarks are made relative to other benchmarks so that surveying can be done relative to the height of one known fundamental benchmark. It could be argued that it isn't really a benchmark unless you can accurately calculate the result based off of a common funda…
Re: Bullshit graph database performance benchmarks
#86Earlier quoted context omitted.
> the final benchmarks were performed on the author's machine No, the author didn't run any benchmarks for Memgraph AFAICT, only for Neo4j. The numbers for mempgraph at the end are from the old benchmark, so on the old hardware.
I don't think that's right. That's not how I understood the article.
The numbers for Memgraph match what is shown on their benchmark website.
Re: Bullshit graph database performance benchmarks
#87If you want to be my hero, find a way to fix this problem: https://maxdemarzi.com/2023/01/09/death-star-queries-in-grap...
Re: Bullshit graph database performance benchmarks
#88Earlier quoted context omitted.
That number should be "The total 3-year price of the entire Priced Configuration must be reported, including: hardware, software, and maintenance charges", so they just took the cost of the hardware used for benchmark, and extended it to 3 years. If you look into the blog post: https://www.databricks.com/blog/2021/11/02/databricks-sets-o... , you will see that it costed $242
Yes the final run, establishing the record, costed $242. I would love to know what the total compute costs for this project was. In real world situations, you run this query daily, or at least multiple times to fine tune it. The point still stands that I can't afford to run on this type of hardware, as it is too expensive, nor do I have such heavy workloads, so these results are not relevant.
Re: Bullshit graph database performance benchmarks
#89Re: Bullshit graph database performance benchmarks
#90Earlier quoted context omitted.
Who in their right mind would implement a database engine in Python?
Absolutely no one, but doesn’t mean you look down on one of the most popular languages as if it’s beneath you.
Weird. Linus Torvalds also hates one of the most popular languages in the world: C++. He must not know anything about operating systems...