MongoDB Performance & Durability • Hacker News Board

Re: MongoDB Performance & Durability

#31

Post by didip » Thu, Jul 08, 2010, 6:05 AM UTC

mikealrogers wrote: post #22

mathias_10gen wrote: post #15

Earlier quoted context omitted.

Actually, that will be a recommended config for replica sets. Most of our slides already show 3 replicas per set. Also, most people have backups so its not really "running naked". See http://www.mongodb.org/display/DOCS/Backups for a few ways to backup mongodb. With LVM/EBS/ZFS or any other snapshotable filesystem, backups can be done almost instantly. With EBS you can even get an insta-slave from the snapshot.

wait, seriously? the suggested default step 1 for MongoDB is to acquire 3 servers? i mean, no other database suggests such a huge default configuration. even knowing that their datacenter can get hit by lightening and all a lot of large production sites don't even run with this kind of redundancy. this seems like a pretty taxing workaround for not keeping an append-only transaction log.

I believe Cassandra also suggest 3 nodes configuration.

Re: MongoDB Performance & Durability

#32

Post by cloudkj » Thu, Jul 08, 2010, 7:16 AM UTC

Actually, I feel that these are some of the reasons that I find MongoDB to be an attractive solution for certain purposes. You can give up some facets of durability and make up for it in ease and performance. MongoDB actually fits quite well into various web apps that have a need for a secondary, non-authoritative datastore that needs to be highly available but not as reliable. Any kind of real-time stream feature would probably benefit from this. Just keep an authoritative copy of your data in your relational datastore of your choice, and asynchronously replicate data as needed to MongoDB. Since it's a non-authoritative datastore, you can denormalize it as much as you need. Works pretty well, IMHO.

Re: MongoDB Performance & Durability

#33

Post by keeto » Thu, Jul 08, 2010, 9:46 AM UTC

jchrisa wrote: post #30

patrickaljord wrote: post #27

Earlier quoted context omitted.

> He did go into several specifics as to why MongoDB might lose your data. He was talking about that post http://www.korokithakis.net/node/119 that doesn't go into any specifics at all. > Nice jab at CouchDB though! But couchdb is slow though. I, like many, switched to mongodb because couchdb was just too slow and when I asked in IRC how to make it faster I was told to run a cluster of couchdb, so, not too different…

For what it's worth, we've managed to make CouchDB about 10x faster (throughput, not latency) in the last year, and we're just getting started on the optimizations.

Care to share how? I'm really interested in what optimizations you did for this.

Re: MongoDB Performance & Durability

#34

Post by [deleted] » Thu, Jul 08, 2010, 10:24 AM UTC

[deleted]

Re: MongoDB Performance & Durability

#35

Post by ekidd » Thu, Jul 08, 2010, 11:22 AM UTC

mjs wrote: post #11

There's a blog post where the MongoDB people explain why they haven't prioritised single server durability: http://blog.mongodb.org/post/381927266/what-about-durability Basically: (a) for real single server durability you need to turn off hardware buffering or have a battery-backed RAID controller to ensure your write really hit disk; (b) this won't help you if your disks fail, and this failure mode is as likely as a…

We run MongoDB on development machines, and they're frequently shut down unexpectedly. But some our development databases take an hour or two to regenerate from scratch, so we prefer to run recovery.

In our experience, MongoDB single-node recovery is very robust. It makes no guarantees of transactional integrity _between_ objects, but in our experience the individual objects have always been recovered intact. According to the MongoDB documentation, recovery will occasionally fail to rebuild objects that span disk pages which were flushed in an inconvenient order, but this will not prevent it from recovering other objects.

So even though MongoDB doesn't have single node durability, and you _really_ ought to run it in replicated mode, it actually manages to have a robust recovery tool.

Re: MongoDB Performance & Durability

#36

Post by kristina » Thu, Jul 08, 2010, 11:34 AM UTC

batasrki wrote: post #19

kristina wrote: post #14

I've tried to post this as a comment on the blog, but it's not showing up (moderated?): ----------- Full disclosure: I work for 10gen. You strategically posted this when my air conditioning was broken, so here are a few thoughts before I go find somewhere cooler. Since CouchDB is "not a competitor" to MongoDB, it's nice of you to put all this time into a public service. > MongoDB, by default , doesn’t actually have a…

Full disclosure: I love MongoDB and use it in a few projects. Having said that, I must ask. Do the "unchecked" writes, configurable fsyncs and multi-node writes exist in the currently stable version of MongoDB or are these features still in alpha or beta? While I do disagree with the article, he clearly pointed at the current version of Mongo.

> Having said that, I must ask. Do the "unchecked" writes, > configurable fsyncs and multi-node writes exist in the > currently stable version of MongoDB or are these features > still in alpha or beta?

They're all available in stable! This might be a documentation fail. It is on the wiki, but suggestions are welcome if you looked at page X and didn't see it.

Re: MongoDB Performance & Durability

#37

Post by kristina » Thu, Jul 08, 2010, 11:37 AM UTC

cbryan wrote: post #21

kristina wrote: post #14

I've tried to post this as a comment on the blog, but it's not showing up (moderated?): ----------- Full disclosure: I work for 10gen. You strategically posted this when my air conditioning was broken, so here are a few thoughts before I go find somewhere cooler. Since CouchDB is "not a competitor" to MongoDB, it's nice of you to put all this time into a public service. > MongoDB, by default , doesn’t actually have a…

"This is because you assume you'll run it on single server. MongoDB's documentation clearly, repeatedly, and earnestly tells people to run MongoDB on multiple servers." Err, what? I'm using MongoDB in production and I've looked at your documentation a bunch. After spending days with your docs open in a browser tab I can't say that it was especially clear on this point. Perhaps I'm particularly ignorant, but I'd wager…

We really, really want people to know they should run on multiple servers. Do you have any suggestions on making it clearer? Where did you look for information about running it in production (so I can add stuff about multiple servers to that page)?

Re: MongoDB Performance & Durability

#38

Post by kristina » Thu, Jul 08, 2010, 11:40 AM UTC

mikealrogers wrote: post #24

kristina wrote: post #14

I've tried to post this as a comment on the blog, but it's not showing up (moderated?): ----------- Full disclosure: I work for 10gen. You strategically posted this when my air conditioning was broken, so here are a few thoughts before I go find somewhere cooler. Since CouchDB is "not a competitor" to MongoDB, it's nice of you to put all this time into a public service. > MongoDB, by default , doesn’t actually have a…

copied from my blog: @kristina for some reason wordpress wanted me to moderate your post so sorry for the delay in it showing up. >> Whoopsy, got your emphasis wrong there ….. Seriously, though, this “unchecked” type >> of write is just supposed to be for stuff like analytics or sensor data, when you’re getting >> a zillion a second and don’t really care some get lost if the server crashes. did the default change? th…

Response (awaiting moderation, I think it's the links?):

-------

> haha, that’s funny. i regularly use non-CouchDB databases and I get along great with > all the people from other databases at conferences.

Oh, I tend to bite people when I find out they use another database . Maybe I should stop that?

> even if i did feel like we were competing, i wouldn’t care. this post really is about > reliability issues i don’t think your users are fully aware of and i honestly hope that > you fix.

You must be thrilled to learn that single server durability is coming. I look forward to a followup post extolling MongoDB’s virtues this fall.

> I responded earlier to the complexity of actually keeping something available that > depends on [multiple servers]. so i won’t cover it again.

Yes, it is a difficult, but not unsolvable, problem. Mongo’s made a bunch of tradeoffs in the awesome vs. easy to program area. For instance, remember last year when CouchDB was saying MongoDB sucked because of its lack of concurrency? That it was too complicated to do concurrency in C++ and that Erlang was the way? Well, now Mongo has concurrency, so on to the next “must have” thing.

>You make it sounds like this is all just a matter of bugs, it’s not, and i find blaming > it on users who don’t use JIRA or get on IRC a little distasteful.

People discuss everything from bugs to architecture to lunch on our various forums. I was trying to say, possibly badly, that we have a lot of ways for people with questions, problems, and suggestions to reach out.

Eliminating the methods I outlined, I’m not sure how people with suggestions could reach the developers, other than telepathy.

Also, the user you cite is far from typical. It sucks that some people don’t like Mongo, but there’s are a lot more out there from those who do: http://codeascraft.etsy.com/2010/07/03/mongodb-at-etsy-part-..., http://blog.eventbrite.com/guest-post-why-you-should-track-p..., http://blog.wordnik.com/what-has-technology-done-for-words-l..., http://www.engineyard.com/blog/2009/mongodb-a-light-in-the-d... and so on.

Re: MongoDB Performance & Durability

#39

Post by ekidd » Thu, Jul 08, 2010, 11:49 AM UTC

mikealrogers wrote: post #22

mathias_10gen wrote: post #15

Earlier quoted context omitted.

Actually, that will be a recommended config for replica sets. Most of our slides already show 3 replicas per set. Also, most people have backups so its not really "running naked". See http://www.mongodb.org/display/DOCS/Backups for a few ways to backup mongodb. With LVM/EBS/ZFS or any other snapshotable filesystem, backups can be done almost instantly. With EBS you can even get an insta-slave from the snapshot.

wait, seriously? the suggested default step 1 for MongoDB is to acquire 3 servers? i mean, no other database suggests such a huge default configuration. even knowing that their datacenter can get hit by lightening and all a lot of large production sites don't even run with this kind of redundancy. this seems like a pretty taxing workaround for not keeping an append-only transaction log.

"the suggested default step 1 for MongoDB is to acquire 3 servers?"

At my previous job, we suffered a catastrophic RAID failure. The controller died while trying repair a degraded array. At this point, we valued the data enough that we weren't going to screw around with replacing the controller and restarting the recovery. The whole RAID array went out to DriveSavers, at the cost of several thousand dollars. Now, DriveSavers are really awesome folks, but my goal is to never do business with them again. :-)

After this incident, our policy was simple: If our data mattered, it had to be replicated.

A 3 server configuration is, admittedly, pretty high end: Even if one node is down, you still have redundancy. You can keep the remaining 2 nodes live while rebuilding the failed node.

I know lots of people who care _deeply_ about data integrity, but who keep their databases on a single server. I find this a bit mystifying: Even the most expensive hardware can die in ugly and unrecoverable ways. And RAID arrays are some of the biggest culprits: Their striping formats are usually undocumented and proprietary.

Re: MongoDB Performance & Durability

#40

Post by po » Thu, Jul 08, 2010, 12:25 PM UTC

wmf wrote: post #28

gfodor wrote: post #16

Ok, so the standard response seems to be "single server durability" isn't supported, but replication makes up for that. How can this be? If no single server guarantees durability of the writes, how can a cluster of those types of machines suddenly cause those writes to be durable? Maybe it's a pedantic argument, but it seems to me that semantically speaking you are simply relying upon luck that your replicated nodes…

They should say that replication makes up for it under the assumption of uncorrelated failures . As you point out, some people may agree with that assumption and some may not.

You're right. I've had a whole datacenter lose power. I am definitely not comfortable with assuming that multiple servers somehow magically makes it safe.

I think the word "durable" implies that the data is written onto a disk. I think of what they are talking about as "redundant". Redundant not-necessarily-durable data.