Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB, Data Durability and Improvements coming in 1.8

paperplanes.de

Post reply on HN
Print view View original

41–50 of 60 posts

Re: MongoDB, Data Durability and Improvements coming in 1.8

#41

Post by steveklabnik »

lwat wrote: post #39
steveklabnik wrote: post #36

Earlier quoted context omitted.

It's good to know that I'm not totally spewing bullshit, and that's the way it should be done. In this case, I would have less than 200 types, and probably under 1000 Foo records total. Updates wouldn't have been a problem. The pain was much bigger than the upside. Especially designing the forms that would end up creating those rows... It's not that it doesn't work. It's that it's not a 'this always fits' solution.

If the amount of data you're storing is so small you should definitely consider de-normalizing it until it's easy enough to work with. When you eventually scale up to millions of Foos you'll probably want to get it back to 3NF. There is nothing wrong with denormalized data in your DB and denormalizing doesn't mean you'd have been better off with some random NoSQL solution. Especially considering that the entire rest…

It's not really random, though. I made a comment elsewhere in this thread, but Mongo is _almost_ a freaking Object database as far as Ruby is concerned. I could do it either way, but doing it with an RDBMs feels like BDSM compared to the loose and fast feel of developing with Mongo.

That said, I always try to use what's right. I don't care how easy it is to store credit card numbers is... I'll just use BrainTree's Vault product instead. ;)

Re: MongoDB, Data Durability and Improvements coming in 1.8

#42

Post by lwat »

bsg75 wrote: post #40
lwat wrote: post #38

Earlier quoted context omitted.

What I'm saying is that I can go up to my SQL server and disconnect the power cord and my database will not be corrupt when I start it back up. Sure if your HDD gets taken out by a meteor then nothing will save you but that's why you have backups.

Not guaranteed. I have had more than one customer experience hardware failure, resulting in a corrupt or suspect SQL Server database, that was unrecoverable via normal means. In each of the cases where the customer had a true standby system, implemented via replication, log shipping, or mirroring, they were able to failover with little (log shipping) or no data loss. In the cases where they had a single, standalone s…

I'm not disagreeing with you, all I'm saying is that SQL Server databases are built from the ground up to resist data corruption and does exceedingly well at it. Not so with NoSQL data stores.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#43

Post by rit »

lwat wrote: post #34
steveklabnik wrote: post #29

Earlier quoted context omitted.

Here's what I posed last time someone asked me this, I got no answer: http://news.ycombinator.com/item?id=1637903

We have exactly the same kind of data relationships in our database. Sure it's complicated and in your example sure you'll end up with a bunch of tables if you normalize it all the way to 3NF. But there's a lot of GOOD STUFF you get when you do, for example if you decide to rename a FooType or a FooSize then you only have to update one record rather than searching every document in your document DB for instances of t…

this is what MongoDBs $rename operator is for...

Re: MongoDB, Data Durability and Improvements coming in 1.8

#44

Post by ericflo »

lwat wrote: post #42
bsg75 wrote: post #40

Earlier quoted context omitted.

Not guaranteed. I have had more than one customer experience hardware failure, resulting in a corrupt or suspect SQL Server database, that was unrecoverable via normal means. In each of the cases where the customer had a true standby system, implemented via replication, log shipping, or mirroring, they were able to failover with little (log shipping) or no data loss. In the cases where they had a single, standalone s…

I'm not disagreeing with you, all I'm saying is that SQL Server databases are built from the ground up to resist data corruption and does exceedingly well at it. Not so with NoSQL data stores.

Not so with MongoDB. Let's please not blame all of NoSQL for MongoDB's bad design decisions.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#45

Post by cdavid »

cdavid wrote: post #20
noelwelsh wrote: post #5

I'm pleased Mongo is getting single server durability. I have never understood why it got so popular without this feature. I'd love to know why people choose Mongo over say, Riak, or CouchDB, as the majority of projects don't need more than one server.

I use mongodb to record fine grained log for performance tests, where it works like a charm. It is fragile (in that I have lost full databases when VM were full, for example) I don't know about riak, but we are forced to use couchdb at work, for the wrong reasons I think, and I was not impressed. First, its performances are pretty bad: document insertion is slow unless you batch them, but even though it is slower tha…

I cannot seem to edit my post, so here is the issue I am referring to w.r.t. couchdb and unused CPU: http://www.mail-archive.com/dev@couchdb.apache.org/msg06520....

here is the interview of Damian Katz concerning couchdb: http://howsoftwareisbuilt.com/2010/06/18/interview-with-dami..., which mentions that large data is not a focus.

There are some companies which seem to use couchdb for large usage, for example bbc (http://enda.squarespace.com/tech/2010/3/4/couchdb-at-scale-4...). I don't know their infrastructure, but they claim to server 4 billions requests as of 4th march 2010 since summer 2009 on a 32 nodes (16 master, 16 backups). Assuming that summer starts in september to get an upper bound of the traffic, this means 250 rq/sec on average, which is nothing impressive for an infrastructure with 32 machines without more information about what they do. Generally, I would not say much about this kind of usecases, but since it is often advertised by couchdb proponents, the burden of the proof is theirs.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#46

Post by va_coder »

noelwelsh wrote: post #5

I'm pleased Mongo is getting single server durability. I have never understood why it got so popular without this feature. I'd love to know why people choose Mongo over say, Riak, or CouchDB, as the majority of projects don't need more than one server.

"I'd love to know why people choose Mongo over say, Riak, or CouchDB"

Here are some reasons:

* excellent documentation

* runs right out of the box

* excellent libraries like Mongoid

* user can easily perform deep queries (e.g person.address.zip = '90901')

* services are available like MongoHQ

Re: MongoDB, Data Durability and Improvements coming in 1.8

#47

Post by lwat »

rit wrote: post #43
lwat wrote: post #34

Earlier quoted context omitted.

We have exactly the same kind of data relationships in our database. Sure it's complicated and in your example sure you'll end up with a bunch of tables if you normalize it all the way to 3NF. But there's a lot of GOOD STUFF you get when you do, for example if you decide to rename a FooType or a FooSize then you only have to update one record rather than searching every document in your document DB for instances of t…

this is what MongoDBs $rename operator is for...

$rename will not magically remove the expense of updating millions of rows in your data store (compared to updating a single row in a normalised DB)

Re: MongoDB, Data Durability and Improvements coming in 1.8

#48

Post by aphyr »

lwat wrote: post #42
bsg75 wrote: post #40

Earlier quoted context omitted.

Not guaranteed. I have had more than one customer experience hardware failure, resulting in a corrupt or suspect SQL Server database, that was unrecoverable via normal means. In each of the cases where the customer had a true standby system, implemented via replication, log shipping, or mirroring, they were able to failover with little (log shipping) or no data loss. In the cases where they had a single, standalone s…

I'm not disagreeing with you, all I'm saying is that SQL Server databases are built from the ground up to resist data corruption and does exceedingly well at it. Not so with NoSQL data stores.

Yeah. There are several nosql backends which offer resistance to corruption under sudden failure. Bitcask, for example, is structured for on-disk data durability using some of the ideas for log-structured filesystems.

Re: MongoDB, Data Durability and Improvements coming in 1.8

#49

Post by jchrisa »

cdavid wrote: post #45
cdavid wrote: post #20

Earlier quoted context omitted.

I use mongodb to record fine grained log for performance tests, where it works like a charm. It is fragile (in that I have lost full databases when VM were full, for example) I don't know about riak, but we are forced to use couchdb at work, for the wrong reasons I think, and I was not impressed. First, its performances are pretty bad: document insertion is slow unless you batch them, but even though it is slower tha…

I cannot seem to edit my post, so here is the issue I am referring to w.r.t. couchdb and unused CPU: http://www.mail-archive.com/dev@couchdb.apache.org/msg06520.... here is the interview of Damian Katz concerning couchdb: http://howsoftwareisbuilt.com/2010/06/18/interview-with-dami... , which mentions that large data is not a focus. There are some companies which seem to use couchdb for large usage, for example bbc (…

I agree we need to improve the interactive query capability of CouchDB.

As far as performance, the key with Couch is to keep the view generation speed faster than the insert rate. So if your users are generating less than about 1000 changes a second, you should be fine on a single server.

More than that and you may need to shard / partition, which can be done in a few different way. The leading option is BigCouch: https://github.com/cloudant/bigcouch

Re: MongoDB, Data Durability and Improvements coming in 1.8

#50

Post by cdavid »

jchrisa wrote: post #49
cdavid wrote: post #45

Earlier quoted context omitted.

I cannot seem to edit my post, so here is the issue I am referring to w.r.t. couchdb and unused CPU: http://www.mail-archive.com/dev@couchdb.apache.org/msg06520.... here is the interview of Damian Katz concerning couchdb: http://howsoftwareisbuilt.com/2010/06/18/interview-with-dami... , which mentions that large data is not a focus. There are some companies which seem to use couchdb for large usage, for example bbc (…

I agree we need to improve the interactive query capability of CouchDB. As far as performance, the key with Couch is to keep the view generation speed faster than the insert rate. So if your users are generating less than about 1000 changes a second, you should be fine on a single server. More than that and you may need to shard / partition, which can be done in a few different way. The leading option is BigCouch: ht…

Actually, the numbers I get are much lower than that (almost two orders of magnitude), but I suspect it is at least partly due to bad usage of couchdb.

I am actually working on simple benchmarks representative of what we do to see if there is something worth submitting as bug issues.

Post reply on HN