Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB, Data Durability and Improvements coming in 1.8

paperplanes.de

Post reply on HN
Print view View original

51–60 of 60 posts

Re: MongoDB, Data Durability and Improvements coming in 1.8

#51

Post by po »

SoftwareMaven wrote: post #31

I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain. Honestly, I would never u…

It was briefly mentioned in the article, but this is exactly how couchdb shuts down:

On-disk, CouchDB never overwrites committed data or associated structures, ensuring the database file is always in a consistent state. This is a “crash-only” design where the CouchDB server does not go through a shut down process, it’s simply terminated.

http://couchdb.apache.org/docs/overview.html

(Please, no mongodb vs. couchdb wars here)

Re: MongoDB, Data Durability and Improvements coming in 1.8

#52

Post by agazso »

po wrote: post #51
SoftwareMaven wrote: post #31

I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain. Honestly, I would never u…

It was briefly mentioned in the article, but this is exactly how couchdb shuts down: On-disk, CouchDB never overwrites committed data or associated structures, ensuring the database file is always in a consistent state. This is a “crash-only” design where the CouchDB server does not go through a shut down process, it’s simply terminated. http://couchdb.apache.org/docs/overview.html (Please, no mongodb vs. couchdb war…

This is exactly how all databases work. Well, except for MongoDB ;)

Re: MongoDB, Data Durability and Improvements coming in 1.8

#53

Post by oomkiller »

mathias_10gen wrote: post #23
richcollins wrote: post #22

You can also use fsync to guarantee that a transaction was written to disk at a severe hit to performance.

Actually that doesn't add any additional durability guarantees in the general case. The only way to use MongoDB durably with a single server is to use the new --dur flag coming in 1.8. I assure you there is no other way.

And how far out is 1.8?

Re: MongoDB, Data Durability and Improvements coming in 1.8

#54

Post by utx00 »

jkkramer wrote: post #16
utx00 wrote: post #11

Earlier quoted context omitted.

what about couchdb?

In my experience, using CouchDB for prototyping is not fun. With SQL and (from what I gather) MongoDB, you can create ad-hoc queries as you go. Creating views in CouchDB requires thought and time -- usually lots of time -- to generate the views, unless you're working with a very small amount of data. Changing your mind is painful. You end up using weird hacks to get around rebuilding your views all the time. Maybe I…

thanks for your feedback.

so far we're going with couch and haven't experienced this. the best part of course is that we can add items to documents on the fly whilst keeping the existing test data! this is major for us. we can go back to testing right away. with MySQL the changes were so major that it was better just dumping and recreating from scratch (we are using django) and this wasn't exactly fast. django has a project "south" that would help with this, but we're eventually deploying to oracle and cannot rely on it or have the time to extend it to properly work with oracle.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#55

Post by megaman821 »

steveklabnik wrote: post #36
lwat wrote: post #34

Earlier quoted context omitted.

We have exactly the same kind of data relationships in our database. Sure it's complicated and in your example sure you'll end up with a bunch of tables if you normalize it all the way to 3NF. But there's a lot of GOOD STUFF you get when you do, for example if you decide to rename a FooType or a FooSize then you only have to update one record rather than searching every document in your document DB for instances of t…

It's good to know that I'm not totally spewing bullshit, and that's the way it should be done. In this case, I would have less than 200 types, and probably under 1000 Foo records total. Updates wouldn't have been a problem. The pain was much bigger than the upside. Especially designing the forms that would end up creating those rows... It's not that it doesn't work. It's that it's not a 'this always fits' solution.

This is what views are for. If it is easier to work with the data denormalized, create a view with a few joins. Some views can even have updates directly applied to them.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#56

Post by steveklabnik »

megaman821 wrote: post #55
steveklabnik wrote: post #36

Earlier quoted context omitted.

It's good to know that I'm not totally spewing bullshit, and that's the way it should be done. In this case, I would have less than 200 types, and probably under 1000 Foo records total. Updates wouldn't have been a problem. The pain was much bigger than the upside. Especially designing the forms that would end up creating those rows... It's not that it doesn't work. It's that it's not a 'this always fits' solution.

This is what views are for. If it is easier to work with the data denormalized, create a view with a few joins. Some views can even have updates directly applied to them.

You're not exactly helping with the "Mongo is significantly less complicated to model my data with" argument. ;)

But still, good to know. I'm bad with views. :[

Re: MongoDB, Data Durability and Improvements coming in 1.8

#57

Post by SoftwareMaven »

po wrote: post #51
SoftwareMaven wrote: post #31

I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain. Honestly, I would never u…

It was briefly mentioned in the article, but this is exactly how couchdb shuts down: On-disk, CouchDB never overwrites committed data or associated structures, ensuring the database file is always in a consistent state. This is a “crash-only” design where the CouchDB server does not go through a shut down process, it’s simply terminated. http://couchdb.apache.org/docs/overview.html (Please, no mongodb vs. couchdb war…

I think that is actually where I read it. It must have resonated internally and became the topic (in my mind).

Re: MongoDB, Data Durability and Improvements coming in 1.8

#58

Post by tessro »

thenduks wrote: post #24
tessro wrote: post #4

This is exciting. MongoDB (and the suite of libraries building up around it) has made prototyping web applications an order of magnitude easier than using SQL. Lack of single-server durability, however, is a showstopper in production, before you've grown enough to justify scaling the database beyond one machine. In my case, that meant going back to MySQL once our schema was finalized (sadly). Looks like that won't be…

What's with the aversion to replicas? I have an app I'm working on launching right now, it's in what you might call 'private beta': real people accessing the app is What I'm saying is that you don't need to grow before adding a second box. My project currently runs on ec2 micro instances and the second box will probably cost you less than your GitHub code hosting costs! Just seems to me that (in ec2 terms, as an exam…

Aren't micro instances $15/month? (Plus EBS backing, unless you want to lose your DB on reboot.)

Regardless, it's added complexity that you can avoid with MySQL early in your application's lifecycle. Even if it's not much more expensive, you have more hardware to administer. Compare that to a Rails+MySQL stack, which you can run on a single box. (There are added benefits of replicas, sure, but they aren't easier to set up, per se, then just installing your stack on one machine when you're getting going.)

Re: MongoDB, Data Durability and Improvements coming in 1.8

#59

Post by tessro »

ericflo wrote: post #10
noelwelsh wrote: post #5

I'm pleased Mongo is getting single server durability. I have never understood why it got so popular without this feature. I'd love to know why people choose Mongo over say, Riak, or CouchDB, as the majority of projects don't need more than one server.

Because they market infinite scalability, insane speeds, and they have a nice API. But people don't realize that, while the speed is fast, you're driving without a seatbelt, and the scalability story is more or less false. They do have a nice API though.

Can you point us toward data that backs up your point that "the scalability story is more or less false"? Seems like FUD.

There are several well-documented, major production deployments of MongoDB already, that seem to contradict your assertion:

    * Etsy
    * CERN
    * BoxedIce (600MM+ documents)
    * BuzzFeed (400MM+ datapoints/month)
That's just a few from http://www.mongodb.org/display/DOCS/Production+Deployments

Re: MongoDB, Data Durability and Improvements coming in 1.8

#60

Post by thenduks »

tessro wrote: post #58
thenduks wrote: post #24

Earlier quoted context omitted.

What's with the aversion to replicas? I have an app I'm working on launching right now, it's in what you might call 'private beta': real people accessing the app is What I'm saying is that you don't need to grow before adding a second box. My project currently runs on ec2 micro instances and the second box will probably cost you less than your GitHub code hosting costs! Just seems to me that (in ec2 terms, as an exam…

Aren't micro instances $15/month? (Plus EBS backing, unless you want to lose your DB on reboot.) Regardless, it's added complexity that you can avoid with MySQL early in your application's lifecycle. Even if it's not much more expensive, you have more hardware to administer. Compare that to a Rails+MySQL stack, which you can run on a single box. (There are added benefits of replicas, sure, but they aren't easier to s…

Yea sorry $15/month (EBS is practically free, as is Elastic IP), but anyway my point still stands at $30/month. I think you'd be hard pressed to find a developer who doesn't spend that much on coffee per month (not to mention merchant accounts and support and code hosting and bug tracking and blah blah blah) :)

As to the rest, sure, it's sys-admin-wise easier to setup. But then you have to consider stuff like database dumps/backups. Where are you storing them? Have you tested your recovery plan? Then there's hassle when you decide you need a dedicated db box and move it there (friday fat-finger, anyone?). True enough that none of these are deal-breaker level issues (evidenced by the ubiquity of MySQL/etc).

With mongo you spend a tiny bit of extra money and a tiny bit of extra time setting up a replica set which handles your backup for you. And then when you need more db horsepower you just start sharding or just adding more replicas and sending reads there. Need to upgrade the db box? Cool just add a huge instance, add it to the replica set and promote it when it's done syncing.

I certainly agree that it's a trade-off... it's just one that I think is not driven by data durability. Either way I need to handle my disaster recovery with some kind of automated system and I'm just suggesting that whether this is some kind of cron-job, restore recipe, etc backup of a db-dump or a replica on a cheap extra box is a wash.

Post reply on HN