MongoDB vs. Clustrix Benchmark • Hacker News Board

Re: MongoDB vs. Clustrix Benchmark

#11

Post by mayank » Mon, Jan 31, 2011, 4:14 PM UTC

moe wrote: post #5

Winning a benchmark against MongoDB on a non-trivial workload is a little bit like winning the special olympics. I'd be more curious to see how Clustrix performs against Cassandra, Riak or HBase in their respective domains. Those seem to be the more serious contenders when it comes to "Big Data".

Would you care to elaborate? Seriously, do you have any pointers or benchmarks for a Mongo vs Cassandra vs X comparison that isn't purely anecdotal, and which takes the relative strengths of each into account in a robust way? A lot of blog posts with performance numbers seem anecdotal or loaded towards one option over the other.

I'm excited by some of the underlying technology in Mongo and its ilk (gossiping protocols, etc.), but there's no doubt that they require different programming techniques than traditional RDBMS. Bit like apples and oranges, isn't it?

EDIT: I noticed you changed RDBMS to "big data". I'm still curious if you have any pointers to fair benchmarks though.

Re: MongoDB vs. Clustrix Benchmark

#12

Post by ajays » Mon, Jan 31, 2011, 4:24 PM UTC

sergei wrote: post #10

psadauskas wrote: post #8

A blog post by the founder, without posting the benchmarks themselves? How can anyone expect to take this seriously?

OK. Fair enough. I'll post the benchmarks.

You can't randomly throw numbers around without listing your benchmark code, the config files, etc.

Secondly: even though it becomes clear eventually, you should mention up front your relationship with Clustrix. Just because you are a founder of Clustrix doesn't necessarily invalidate your findings, but full disclosure is always a good idea.

Re: MongoDB vs. Clustrix Benchmark

#13

Post by ozataman » Mon, Jan 31, 2011, 4:30 PM UTC

Another point to notice: He seems to have run his benchmarks under very trivial data-schemas with a single/simple table. A big (speed, simplicity) advantage of No-SQL is the ability to embed lots of data within the parent model and manage a single table where you would have to manage many in a SQL database.

I would be very interested to see a comparison where a large and "real" data model (that contains 6-7 "joined" tables for the SQL setup and a single table with the embedded document model for the No-SQL setup) is injected into each of the technologies.

Also, it is just horrible style not to include the benchmark code for peer review. Delivers near-0 credibility.

Re: MongoDB vs. Clustrix Benchmark

#14

Post by jhugg » Mon, Jan 31, 2011, 4:34 PM UTC

The attacks on NoSQL seem a little harsh to me. People needed scalable systems. They needed them ASAP. Nobody in the RDBMS camp was even hinting at products targeting this market. Then, the NoSQL camp built systems that scale (albeit with compromises).

The fact that this has spurred the others to start building scalable RDBMSs is great. But let's not pretend these new RDBMSs won't have compromises, they'll just have different ones.

The important thing is for developers to make smart decisions about what tradeoffs make the most sense for scaling their application. Different applications will require different tradeoffs.

Disclaimer: I work for VoltDB.

Re: MongoDB vs. Clustrix Benchmark

#15

Post by jbellis » Mon, Jan 31, 2011, 4:40 PM UTC

wmf wrote: post #2

Price is the elephant in the room here; NoSQL exists because people aren't willing to pay for real databases. Building yet another expensive (i.e. > $0) database that doesn't even work in the cloud won't help the Web 2.0 crowd.

Well, that's partly right. Yes, clustrix is more expensive than the open-source databases.

But, companies like Twitter and Netflix certainly have the budget for Exadata (which is what clustrix wants to be when it grows up); they are using Cassandra instead not just because it scales on commodity hardware (the other price factor besides licensing) but also because it works across multiple datacenters which two-phase commit systems can't do no matter how big your budget is, and because its availability (failure tolerance) model is much more robust.

(Many NoSQL systems don't provide these advantages either, which is why lumping all non-relational systems together is usually not helpful.)

Re: MongoDB vs. Clustrix Benchmark

#16

Post by jnewland » Mon, Jan 31, 2011, 4:42 PM UTC

has anyone out there actually used or evaluated clustrix? i haven't been able to find anything on the internet about this thing that hasn't come straight from the company.

Re: MongoDB vs. Clustrix Benchmark

#17

Post by jbellis » Mon, Jan 31, 2011, 4:47 PM UTC

bryanmig wrote: post #7

So a guy who is an expert in Clustrix (and knows how to setup, tune, etc) compares it against some other technology that he does not know (and does not know how to setup, tune, etc) and comes to the surprising realization that his technology is better? Where have I seen this before? Oh right.. every time I see "Technology A vs Technology B" comparisons. Naturally his results are in his favor, otherwise he would not h…

In fairness, you don't have to tune MongDB poorly (deliberately or otherwise) to get poor performance with a workload involving substantial numbers of writes; it's well-documented that there is a global lock (http://www.mongodb.org/display/DOCS/How+does+concurrency+wor...) that prevents reads during write operations.

That said, there are certainly nosql systems with better scaling and concurrency stories* than mongodb out there that he could have benchmarked against. :)

*I'm a cassandra committer

Re: MongoDB vs. Clustrix Benchmark

#18

Post by antirez » Mon, Jan 31, 2011, 4:53 PM UTC

In this article MongoDB == NoSQL, that is not the case. Different NoSQL solutions have different use cases. Also IMHO MongoDB is pretty SQLish in the data model, so you are actually comparing two implementations of a similar data model here, and one may be superior to the other one or the other way around I guess. No surprise.

A more interesting attempt is IMHO to check how the difference in the data model of some NoSQL solution can lead to very different performances.

For instance Clustrix VS Redis can be interesting. Examples:

1) A lot of writes against a table where you require then to get things ordered by insertion time. With Redis is is just LPUSH + LRANGE. Try to do a read/write test where many clients are writing and reading at the same time (real world), against a table (or Redis list) with millions of elements.

2) Range queries when there are a lot of writes against this indexes. For instance a table with a score (we are modeling an online game leaders board), a lot of inserts of new scores. Get ranges between random intervals at the same time. Again, many clients writing, many reading.

Re: MongoDB vs. Clustrix Benchmark

#19

Post by moe » Mon, Jan 31, 2011, 4:57 PM UTC

mayank wrote: post #11

moe wrote: post #5

Winning a benchmark against MongoDB on a non-trivial workload is a little bit like winning the special olympics. I'd be more curious to see how Clustrix performs against Cassandra, Riak or HBase in their respective domains. Those seem to be the more serious contenders when it comes to "Big Data".

Would you care to elaborate? Seriously, do you have any pointers or benchmarks for a Mongo vs Cassandra vs X comparison that isn't purely anecdotal, and which takes the relative strengths of each into account in a robust way? A lot of blog posts with performance numbers seem anecdotal or loaded towards one option over the other. I'm excited by some of the underlying technology in Mongo and its ilk (gossiping protocol…

Well, don't trust a benchmark that you haven't faked yourself they say.

I was just trying to point out that MongoDB is too easy a target here. The problems it has under high load are fairly well-known, at least to anyone who tried to benchmark it outside of their MacBooks. Just bulk-load a couple million records and watch it tip over if you don't believe me - I'm not making it up and neither is Sergei.

However if Clustrix wants to impress with benchmarks then they should pick an equal opponent. MongoDB is not exactly relevant for companies that consider the calibre (and cost!) of Clustrix.

Re: MongoDB vs. Clustrix Benchmark

#20

Post by richardmarr » Mon, Jan 31, 2011, 5:07 PM UTC

Sergei,

I think it was a mistake to put this post on a blog with no other content. I think it leaves the reader with the impression that this is the only thing you want to contribute to the community... and as other commenters have pointed out that contribution could be perceived negatively.

Something to think about next time.