MongoDB Performance & Durability

Scary.

I'm a big fan of MongoDB, and I think its replicated durability characteristics are good enough for many classes of applications. If you could lose a minute of data and have it just be "bad" instead of "customer enraging and business threatening", then it's a very nice database system.

However--I do think the decision to make writes "fire and forget" is just a mistake. If you use abstractions like a connection pool under heavy concurrency, you can get unpredictable behavior in terms of when the data is "actually there."

For example, all in one thread:

with connection pool: do_write operation A

do other things...

with connection pool: read something, assuming A has been applied

Specifically, the fact that you get an arbitrary connection out of the pool means you cannot be sure that the database has completely processed operation A before executing your new query.

MongoDB has a "safe=" flag in their Python bindings that implements the project's official (http://www.mongodb.org/display/DOCS/Last+Error+Commands#Last...) recommendation for "it is written" consistency. It's a bit of a hack, but it calls "getLastError()" on the connection that does the write before returning from the update()/save()/insert()/delete() call. I think it's astonishing this behavior isn't the default.

In fact, I recently updated diesel's bindings to be safe=True, so getLastError() is always executed on write operations to make sure they succeeded before the call returns:

http://github.com/jamwt/diesel/commit/95cd71d82ffc5c308060b6...

It's not really about losing a minute of data though--the whole thing could become corrupted, and repairing it will be extremely difficult.

Here we go again. Getting a bit tiresome hearing about this "durability issue".

I really don't understand the problem. No, MongoDB isn't durable the same way MyISAM/InnoDB is with MySQL.

But I think that is clear as day, certainly no news to me when choosing MongoDB.

Been running it in production for 3 months, working fine. If it for some reason would be data loss or corruption, I wouldn't come crying to the devs. I know what risks I took when making the decision.

If the backups aren't there I can only blame myself when such event happen.

Clustered durability. You can request, per-write, "don't return until this data has been persisted to N other servers," with N chosen per-write.

So if you're running master/slave and choose N to be 1, you're durable (at the two server level). Run 5-replica sets and choose N to be 3 or 4, and you're basically guaranteed durability. One of your nodes goes down hard and bad things happen? You have four additional clones of it sitting and waiting to be copied.

This is actually more durable in cases of extreme hardware failure like, say, your RAID controller going out. However, it requires that you spend more on hardware. So do several other things Mongo does, like using a lot of RAM and disk to get faster writes, so that's in keeping with a lot of their other design decisions.

It's not perfect for every project, but it's a great choice for most of the same projects where you'd use Mongo in the first place.