Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB, Data Durability and Improvements coming in 1.8

paperplanes.de

Post reply on HN
Print view View original

31–40 of 60 posts

Re: MongoDB, Data Durability and Improvements coming in 1.8

#31

Post by SoftwareMaven »

I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain.

Honestly, I would never use a database that doesn't make safeguarding my data as its number one priority. Even if I'm just dumping logs in it, the time I really need to know what's going on is likely the time those safeguards are put to the test.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#32

Post by steveklabnik »

SoftwareMaven wrote: post #31

I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain. Honestly, I would never u…

Are you thinking of Fail-fast software? http://en.wikipedia.org/wiki/Fail-fast

Re: MongoDB, Data Durability and Improvements coming in 1.8

#33

Post by wmf »

SoftwareMaven wrote: post #31

I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain. Honestly, I would never u…

http://www.usenix.org/events/hotos03/tech/full_papers/candea...

Re: MongoDB, Data Durability and Improvements coming in 1.8

#34

Post by lwat »

steveklabnik wrote: post #29
lwat wrote: post #28

Earlier quoted context omitted.

"Sometimes, data doesn't fit a relational model well." I keep hearing this but I've yet to see any good examples.

Here's what I posed last time someone asked me this, I got no answer: http://news.ycombinator.com/item?id=1637903

We have exactly the same kind of data relationships in our database. Sure it's complicated and in your example sure you'll end up with a bunch of tables if you normalize it all the way to 3NF. But there's a lot of GOOD STUFF you get when you do, for example if you decide to rename a FooType or a FooSize then you only have to update one record rather than searching every document in your document DB for instances of those names so you can rename them. Hell if you have many documents and 24/7 clients this may become IMPOSSIBLE to do while still guaranteeing consistency in your data.

Your example is not that unusual or even that complicated. What is about it that doesn't work on a RDBMS? You can't just say the solution sucks because it uses more tables that you feel it should.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#35

Post by ethangunderson »

megaman821 wrote: post #8

I am never sure what niche MongoDB is supposed to fill. If I want a large cluster to handle "big data" Riak or Cassandra seem to fit the bill better. If I want speed Redis is great. If I want a schema-less SQL-like (but not SQL) database, MongoDB?

Just like most things, it's not that simple. For example...

Cassandra is good for 'big data' if you're ok dealing with the repercussions of eventual consistency, and questionable performance characteristics. Not to mention the mediocre community support.

Redis is a fantastic choice, if you're ok with a limited data model and no diskstore option(coming soon though!)

Picking a datastore that fits your data model, access patterns, support needs, etc. can be a daunting task, and simplifying it to this degree just leaves too much out of the equation.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#36

Post by steveklabnik »

lwat wrote: post #34
steveklabnik wrote: post #29

Earlier quoted context omitted.

Here's what I posed last time someone asked me this, I got no answer: http://news.ycombinator.com/item?id=1637903

We have exactly the same kind of data relationships in our database. Sure it's complicated and in your example sure you'll end up with a bunch of tables if you normalize it all the way to 3NF. But there's a lot of GOOD STUFF you get when you do, for example if you decide to rename a FooType or a FooSize then you only have to update one record rather than searching every document in your document DB for instances of t…

It's good to know that I'm not totally spewing bullshit, and that's the way it should be done.

In this case, I would have less than 200 types, and probably under 1000 Foo records total. Updates wouldn't have been a problem. The pain was much bigger than the upside. Especially designing the forms that would end up creating those rows...

It's not that it doesn't work. It's that it's not a 'this always fits' solution.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#37

Post by harryh »

lwat wrote: post #19
harryh wrote: post #17

Earlier quoted context omitted.

Because single server durability is a myth when hardware can fail at any time.

We're talking about data durability. When my SQL server crashes due to hardware failure I know that when I eventually get it back up and running the data will be consistent with at worst the last couple of transactions being rolled back.

This is most certainly not true. Disks fail in ways where the whole volume becomes unreadable all the time.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#38

Post by lwat »

harryh wrote: post #37
lwat wrote: post #19

Earlier quoted context omitted.

We're talking about data durability. When my SQL server crashes due to hardware failure I know that when I eventually get it back up and running the data will be consistent with at worst the last couple of transactions being rolled back.

This is most certainly not true. Disks fail in ways where the whole volume becomes unreadable all the time.

What I'm saying is that I can go up to my SQL server and disconnect the power cord and my database will not be corrupt when I start it back up. Sure if your HDD gets taken out by a meteor then nothing will save you but that's why you have backups.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#39

Post by lwat »

steveklabnik wrote: post #36
lwat wrote: post #34

Earlier quoted context omitted.

We have exactly the same kind of data relationships in our database. Sure it's complicated and in your example sure you'll end up with a bunch of tables if you normalize it all the way to 3NF. But there's a lot of GOOD STUFF you get when you do, for example if you decide to rename a FooType or a FooSize then you only have to update one record rather than searching every document in your document DB for instances of t…

It's good to know that I'm not totally spewing bullshit, and that's the way it should be done. In this case, I would have less than 200 types, and probably under 1000 Foo records total. Updates wouldn't have been a problem. The pain was much bigger than the upside. Especially designing the forms that would end up creating those rows... It's not that it doesn't work. It's that it's not a 'this always fits' solution.

If the amount of data you're storing is so small you should definitely consider de-normalizing it until it's easy enough to work with. When you eventually scale up to millions of Foos you'll probably want to get it back to 3NF.

There is nothing wrong with denormalized data in your DB and denormalizing doesn't mean you'd have been better off with some random NoSQL solution. Especially considering that the entire rest of your DB is still very much relational data so what happens to all of that when you dump Postgre for Mongo?

Re: MongoDB, Data Durability and Improvements coming in 1.8

#40

Post by bsg75 »

lwat wrote: post #38
harryh wrote: post #37

Earlier quoted context omitted.

This is most certainly not true. Disks fail in ways where the whole volume becomes unreadable all the time.

What I'm saying is that I can go up to my SQL server and disconnect the power cord and my database will not be corrupt when I start it back up. Sure if your HDD gets taken out by a meteor then nothing will save you but that's why you have backups.

Not guaranteed. I have had more than one customer experience hardware failure, resulting in a corrupt or suspect SQL Server database, that was unrecoverable via normal means.

In each of the cases where the customer had a true standby system, implemented via replication, log shipping, or mirroring, they were able to failover with little (log shipping) or no data loss.

In the cases where they had a single, standalone server, the option was to restore the last known good backup, or sent the database files had to Microsoft for analysis and repair.

ANY system (RDBMS, NoSQL, or otherwise), should have a standby replica to prevent data loss. If you data is stored on a single machine, you are doing it wrong.

Post reply on HN