MongoDB vs. Clustrix Benchmark

Entirely agree with you, eventual consistency and quorums protocols all have their drawbacks. Difficult of developing against that (and the natural reaction of just "handwaving" the consistency model rather than using optimistic locking with a vector clock at the identified synchronization points) is the main one. I have, however, also seen the situation where developers would fail to understand how to correctly deal with availability and build ad-hoc, poorly thought-out HA layers on top of non-HA systems. The typical memcache + MySQL deployment is the prime example of that: there's neither the high availability, nor an understandable consistency/cache coherence model.

> Note that his "right answer" for performance (not availability, so it's a bit of an aside from your comment) is to dynamically repartition, which is what Clustrix does. Note that we can actually do this type of repartitioning on just the "hot" data, using MVCC to avoid any downtime (unlike Mongo, which blocks all writes).

Dynamic repartitioning is also what HBase/BigTable do ("tablet splitting"). The MVCC approach is interesting way around the unavailability window, by the way.

> By the way, the Google Megastore project I believe you're referring to is not actually eventually consistent in the same way that, say, Cassandra is. Megastore is fully ACID within an entity group, meaning that lowering eventually consistency down to the per-node level was not the solution that Google went with. BigTable is also not eventually consistent.

Yes, I did not mean that Megastore is eventually consistent. It is not, it's ACID compliant and uses Paxos. Real question: was this not stated clearly in my comment (I'd like to edit to clarify if it is)?

Megastore, however, is an example of a "NoSQL" system and I meant to use it as proof that not all "NoSQL" systems give up consistency: ACID is orthogonal to the query language.

Quite frankly I find the whole query language debate (and "NoSQL" marketing gimmick) to be quite pointless, the distributed systems and systems programming aspects are a lot more interesting to me.

I can scale PostgresSQL pretty well, thanks. PostgreSQL is used on some of the biggest databases in the world, but no one's going to tell you which. :)

Your benchmark is pretty much irrelevant because Clustrix sells hardware appliances, not the software standalone.

Edit: That's not to say MongoDB didn't fail your benchmark. But the issues you highlight (single mutex) are known. Test against a real database and your results don't look that impressive for a 10 node setup.

All that said, Clustrix looks great. Just make it available sans-hardware.

If I want to trial Oracle or Clustrix, well, I'm going to have to start a contact with a sales organization and then they need to pre-qualify me, and then they need to qualify me and then I'll spend a few hours on the phone talking to people (which might take a few weeks in wall-clock time.)

If Clustrix offered cloud-based deployment of their database, they could have a small free tier for people to play around with. Add a console applicaiton for accessing the API. Make it really developer friendly.

Imagine if you could just type:

    $ sudo gem install clustrix
    $ clustrix create my-test-app
    Created cloud.clustrix.com/my-test-app
    username: my-test-app
    password: xd634shx

If they had that, I'd be trying it out in a flash. Then what if they had a per-usage pricing system like AWS? Something where you pay per gigabyte. Something that allows you to test stuff out for a few dollars a mount, and then when you're ready, scale it right up to a few thousand or more.

So relational data is hard but implementing appropriate consistency checks in your application is easy? Or do you just skip that second part and hope for the best?

> This is why Cassandra is now offering counters as an API feature

This is specific to Cassandra, where the development team chose not to implement optimistic locking via version vectors. That said the counters patch essentially implements a vector of counters, very similar conceptually to a vector clock.

That said, there are some scenarios where quorums (in absence of an agreed upon view of the cluster e.g., a zookeeper-based failure detectors) could mean concurrent vector clocks, which means it's up to the application to reconcile it.

If your application can not reconcile a split brain scenario on its own and depends on a total order then yes, it's not a good fit for EC.

As I've said, I also think that serializable consistency is easier for applications to deal with (they can assume a total order) than eventual consistency but provides the strong benefit of read availability (something, quite frankly, most Internet applications require).

Most complex applications have points where consistency can be relaxed (either somewhat or fully) and points where consistency is needed.

In "we chose consistency" and "we chose availability" cases, the applications will have to deal with the consequences of those choices: much as developers may handwave the implications of an eventually consistent system (since it returns right results most of the time, not thinking whether or not their application is sensitive to a non-serializable order) they might also implement ad-hoc, poorly thought out HA solutions on top of strong consistent systems -- often times losing both availability and consistency.

The typical usage scenario of masking MySQL failures and latency by using memcache comes to mind. VoltDB is very correct to remove the need to use memcached for read latency (a caching layer in front of a database which already a cache which sits on top of a file system with a cache is a caching layer too many), but in many cases I've seen memcached used to accept reads (and sometimes writes) while the underlying database is unavailable.

Finally, the whole debate about atomicity is quite meaningless if (like many applications) you're fronting your data access layer with an RPC framework (your RPC call may go through to a database and perform a write even it times out to the application i.e., implying non-atomicity).

The CALM paper is a big move into the right direction: being able to determine synchronization points at which ACID transactions (or weaker forms of synchronizations if permissible e.g., quorums and version vectors) can be used. Incidentally, I also think a declarative language (whether like Bloom/Bud or closer to SQL) is a good way to express the constraints and let a system (which may not be all that different from a query planner) determine whether those points occur.