Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“The benchmark numbers are completely wrong for both databases”

github.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: “The benchmark numbers are completely wrong for both databases”

#21

Post by weddpros »

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that.

I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at least he tried... and the author has corrected the discrepancies since then.

Now imagine the benchmark was against [your favorite DB here] with even stronger results against RethinkDB. Notice how the most upvoted comment is joking about MongoDB. The second one is a pro-mysql comment. What's the point? Would it have been a better benchmark if it read "mysql is 10x faster than RethinkDB?" or "MongoDB is even slower than RethinkDB"?

Re: “The benchmark numbers are completely wrong for both databases”

#23

Post by sarciszewski »

weddpros wrote: post #21

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at le…

I have to say, it's rare for me to agree with a HN comment as much as I do to this one. Well said!

Re: “The benchmark numbers are completely wrong for both databases”

#24

Post by sklivvz1971 »

weddpros wrote: post #22
sklivvz1971 wrote: post #18

This old comic piece seems to still apply: http://www.mongodb-is-web-scale.com/

Then how on earth is it possible to build Parse.com with MongoDB? http://blog.parse.com/announcements/mongodb-rocksdb-parse/

Maybe you didn't get the joke. The joke is not about MongoDB, but about MongoDB fanbois that care only about some very narrow definition of "performance".

The wider message is that DBs are way more complex beasts that is meaningful to test this way.

Obviously there are cases in which MongoDB is a great choice, but equally obviously tests like this should not be a reason for the choice.

Re: “The benchmark numbers are completely wrong for both databases”

#25

Post by sklivvz1971 »

weddpros wrote: post #21

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at le…

I suppose the author of the test forgot the "only test realistic scenarios" best practice.

Re: “The benchmark numbers are completely wrong for both databases”

#26

Post by weddpros »

sklivvz1971 wrote: post #24
weddpros wrote: post #22

Earlier quoted context omitted.

Then how on earth is it possible to build Parse.com with MongoDB? http://blog.parse.com/announcements/mongodb-rocksdb-parse/

Maybe you didn't get the joke. The joke is not about MongoDB, but about MongoDB fanbois that care only about some very narrow definition of "performance". The wider message is that DBs are way more complex beasts that is meaningful to test this way. Obviously there are cases in which MongoDB is a great choice, but equally obviously tests like this should not be a reason for the choice.

I do understand that second degree, but I don't think the author is a fanboy...

At work, we're also Mongodb users. If tomorrow we try to benchmark against Cassandra, performance will probably be the selling point. I don't think it's absurd to compare mongodb and rethinkdb, they're very similar dbs.

As for the benchmark, I agree it doesn't explore every facet of both databases and focus on performance... That's what benchmarks do.

The intent of the author may have been to challenge his existing choice (mongodb) which is a good thing. The (corrected) results may lead to: expect no performance gain if we migrate to RethinkDB... What's wrong with that?

Re: “The benchmark numbers are completely wrong for both databases”

#27

Post by threeseed »

tinussky wrote: post #11
threeseed wrote: post #10

Earlier quoted context omitted.

Everyone likes to see how their tool of choice performs against other tools. And it's an important (albeit just one part) of deciding which product to use.

I always say do your own benchmarking for your own use case. The risk of a colored benchmark is quite high when benchmark is done by owner of product or by "fan" of product. With the exception of a well explained, clear benchmark that everyone can understand and reproduce easily.

But in this case the author has a public Github page with the code used to recreate the benchmark.

So I don't see how any claim of bias applies here.

Re: “The benchmark numbers are completely wrong for both databases”

#28

Post by sklivvz1971 »

weddpros wrote: post #26
sklivvz1971 wrote: post #24

Earlier quoted context omitted.

Maybe you didn't get the joke. The joke is not about MongoDB, but about MongoDB fanbois that care only about some very narrow definition of "performance". The wider message is that DBs are way more complex beasts that is meaningful to test this way. Obviously there are cases in which MongoDB is a great choice, but equally obviously tests like this should not be a reason for the choice.

I do understand that second degree, but I don't think the author is a fanboy... At work, we're also Mongodb users. If tomorrow we try to benchmark against Cassandra, performance will probably be the selling point. I don't think it's absurd to compare mongodb and rethinkdb, they're very similar dbs. As for the benchmark, I agree it doesn't explore every facet of both databases and focus on performance... That's what b…

Well, OK. I am not calling the author a fanboy, and i'll agree they probably aren't.

There are at least 3 things here which invalidate this post because they make the numbers uncomparable:

1. Is the hardware adequate to run the tests, or does it favor one database?

2. Are both databases tuned to the use case?

3. Running only one type of action at a time is meaningless. Contention between reads and writes is what normally drives performance stats.

Re: “The benchmark numbers are completely wrong for both databases”

#29

Post by weddpros »

sklivvz1971 wrote: post #25
weddpros wrote: post #21

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at le…

I suppose the author of the test forgot the "only test realistic scenarios" best practice.

We know nothing about the author's use case...

Maybe he wanted to know where each DB shines compared to each other, to see if some workloads are better suited to one or the other.

Of course, benchmarks "should" include concurrent reads/updates/writes/deletes because it can make a huge difference depending on the DB's implementation.

Of course, the author "should" also have tested sharding / durability / resistance to partition / resource consumption in his tests... Maybe he didn't have the resources to test properly. I also do quick&dirty benchmarks like these, mostly because exhaustive benchmarks cost so much more (time, money, expertise)...

Re: “The benchmark numbers are completely wrong for both databases”

#30

Post by timmaxw »

weddpros wrote: post #21

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at le…

> the author has corrected the discrepancies since then

As of the time I posted this comment, the blog post still seems to be comparing indexed MongoDB operations against non-indexed RethinkDB operations. Under those conditions I'd expect RethinkDB to be at least 1000x slower than MongoDB. The fact that he's finding that RethinkDB is only 3x slower than MongoDB makes me think that there are still other major problems with this benchmark.

> No, but at least he tried...

It's true that the author tried; but that doesn't change the fact that people are going to read this blog post and assume that the numbers are at least approximately correct. As a RethinkDB employee, it really frustrates me to see RethinkDB being judged according to benchmarks that are conducted so carelessly that they are essentially random.

I think this is the fourth time in the past year that I've seen a third party try to benchmark RethinkDB and get something wrong. Maybe we need to start a "best practice" of checking in with the maintainers of a project before publishing benchmark results about the project.

Post reply on HN