Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“The benchmark numbers are completely wrong for both databases”

github.com

Post reply on HN
Print view View original

41–50 of 58 posts

Re: “The benchmark numbers are completely wrong for both databases”

#41

Post by userbinator »

lukev wrote: post #37

This is why a lot of commercial databases have a "you may not publish benchmarks" clause in the license. It seems unfair and restrictive, but benchmarking is hard , and even where users get everything else right every data load is different. It's easy to see why companies don't want writeups like this one dominating searches for " performance".

It seems even more unfair and restrictive to do that.

If you prohibit benchmarks, it's essentially saying "we have something to hide" (like bad performance...), or "we don't want competition".

I think it's far better to point out the flaws and specifics than to censor any attempt at comparison.

Re: “The benchmark numbers are completely wrong for both databases”

#42

Post by mikecmpbll »

webtards wrote: post #8

Not sure what's worse here - people relying on third party benchmarks (hint: always do your own; see how a tool performs on your data, on your hardware, for your problem set), or the fanboy-ish panic when they are unsettled that a benchmark might make their chosen toy less shiny?

The former :)

Re: “The benchmark numbers are completely wrong for both databases”

#43

Post by DannoHung »

weddpros wrote: post #21

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at le…

> "do your own benchmark" best practice

Is this a best practice? It seems like we've been delivered evidence that it is really hard to do good benchmarks unless you're already intimately familiar with what you're testing, which says something about how hard it is to make a good choice.

I don't know about other industries, but this sort of result is what stuff like the STAC M3 Benchmark suite was designed for: Typical usecases that experts can implement so you can get realistic performance comparisons.

Re: “The benchmark numbers are completely wrong for both databases”

#44

Post by copsarebastards »

weddpros wrote: post #21

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at le…

So basically you're saying that because he cares and because he tried, and because the comments were dumb, nobody should criticize him? Do you really not care about getting accurate results?

Running benchmarks is an engineering practice. If you failed to get meaningful results, you failed. Yes, he cares, yes, he tried, yes, the comments are dumb, but he still failed. Sure, I'll give the guy kudos for trying, but I'm not going to pretend he didn't fail. As far as I'm concerned, telling someone they failed is a favor, because now they can change their methodology, try again, and maybe succeed. It's part of the process of achieving meaningful results. The entire point of what he's doing is to achieve meaningful results, not to get a participation medal.

Your response reminds me of this: https://www.youtube.com/watch?v=gSjLiQxEZlM

Re: “The benchmark numbers are completely wrong for both databases”

#45

Post by insaneirish »

JDDunn9 wrote: post #20

Would be nice to see some official benchmarks from RethinkDB. As this illustrates, there aren't any good ones out there.

> Would be nice to see some official benchmarks from RethinkDB.

Of some artificial scenario that people will claim does not reflect [their] reality?

> As this illustrates, there aren't any good ones out there.

The only good benchmarks are the ones that you define to be representative of something you'll actually see in reality.

Re: “The benchmark numbers are completely wrong for both databases”

#46

Post by mikecmpbll »

Any output like this, unless maliciously fallacious, is contributing in some way to the general understanding of the software concerned and benchmarking best-practices, even through its mistakes.

It's the job of the reader to judge their sources wisely, and interrogate what they read, rather than the job of the author to conduct their explorations in private.

Understandably, it can be frustrating for people involved in the projects but that's just the nature of the beast. They can do things to help their cause by championing good examples of benchmarking, even those which don't look upon them favourably.

Re: “The benchmark numbers are completely wrong for both databases”

#48

Post by JDDunn9 »

copsarebastards wrote: post #44
weddpros wrote: post #21

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at le…

So basically you're saying that because he cares and because he tried, and because the comments were dumb, nobody should criticize him? Do you really not care about getting accurate results? Running benchmarks is an engineering practice. If you failed to get meaningful results, you failed. Yes, he cares, yes, he tried, yes, the comments are dumb, but he still failed. Sure, I'll give the guy kudos for trying, but I'm…

You must be fun at parties...

Re: “The benchmark numbers are completely wrong for both databases”

#49

Post by igouy »

amaranth wrote: post #19
jakozaur wrote: post #4

Benchmarking is hard and a lot of reports are bogus. However they are still very useful for a lot of developers. Benchmarking programming languages got better. E.g.: http://benchmarksgame.alioth.debian.org/ gives roughly idea about performance of programing languages. I wish something similar existed for databases. I think exact figures would be hard to get, but I believe there are many 2x 10x differences that we sho…

Don't a lot of those benchmarks end up only measuring how fast your language can call out to GMP to do the real work? And regex-dna ends up measuring your regex implementation which for a lot of them is again just going to be measuring how fast they can all call out to PCRE. They're neat and all and it is called the benchmarks game but I wish they'd remove the ones that end up getting gamed like that.

>> Don't a lot of those benchmarks… No.

>> …how fast they can all call out to PCRE. No. Have you looked at the programs?

Re: “The benchmark numbers are completely wrong for both databases”

#50

Post by mejari »

userbinator wrote: post #41
lukev wrote: post #37

This is why a lot of commercial databases have a "you may not publish benchmarks" clause in the license. It seems unfair and restrictive, but benchmarking is hard , and even where users get everything else right every data load is different. It's easy to see why companies don't want writeups like this one dominating searches for " performance".

It seems even more unfair and restrictive to do that. If you prohibit benchmarks, it's essentially saying "we have something to hide" (like bad performance...), or "we don't want competition". I think it's far better to point out the flaws and specifics than to censor any attempt at comparison.

> it's essentially saying "we have something to hide" (like bad performance...), or "we don't want competition"

This seems a totally imaginary dichotomy when literally the comment you're replying to presents an alternative option, namely; "We don't want you to publish things that are almost always going to be wrong and misleading"

Post reply on HN