Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

11–20 of 153 posts

Re: A Year with MongoDB

#11

Post by antirez »

LeafStorm wrote: post #5
jrussbowman wrote: post #3

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scalin…

While I agree that you should definitely use your tools in the best way you're capable of, I think for most people there's a baseline expectation that if you save data in a database, that data will be safe (at the very least, recoverable) unless something happens like the server catching on fire. Nearly every other major database has this as the default -- MySQL, PostgreSQL, CouchDB, and Berkeley DB to name a few. (R…

You are wrong about Redis. Cheers

Re: A Year with MongoDB

#13

Post by sirn »

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers.

Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?)

I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"

Re: A Year with MongoDB

#14

Post by jbellis »

rdtsc wrote: post #10

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability…

According to http://www.mongodb.org/display/DOCS/getLastError+Command it is still unsafe by default.

Re: A Year with MongoDB

#15

Post by mitchellh »

sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?)

Yes, we did on-the-fly migration as we loaded old data in.

Doing full data migration was not really an option because querying from MongoDB on un-indexed data is so slow, and paging in all that data would purge hot data, exacerbating the problem.

> I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"

That's exactly what happens.

Re: A Year with MongoDB

#16

Post by madworld »

paulsutter wrote: post #4

This article is an excellent articulation of the strengths and (fixable) issues with mongoDB. I like MongoDB a lot, and the improvements suggested would really strengthen the product and could make me more comfortable to use it in more serious applications.

How is the global write lock "fixable" without a major rewrite of the codebase?

Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.

Re: A Year with MongoDB

#17

Post by zzzeek »

mitchellh wrote: post #7
jrussbowman wrote: post #3

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scalin…

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark sin…

typo in the numbers ? I can get your mongoDB number but not pg:

    >>> seconds_per_month = 60 * 60 * 24 * 30
    >>> ops_per_month_pg = 1000 * seconds_per_month
    >>> ops_per_month_mg = 200 * seconds_per_month
    >>> 330.0 / ops_per_month_pg * 100
    1.2731481481481482e-05
    >>> 330.0 / ops_per_month_mg * 100
    6.36574074074074e-05

Re: A Year with MongoDB

#18

Post by saurik »

jrussbowman wrote: post #3

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scalin…

The behavior described in the article, though, is just a fundamental misuse of the hardware resources. If I'm hitting write bottlenecks, I need to shard: that's just a fact of reality. However, write throughput and read throughput should be unrelated (as durable writes are bottlenecks by disk and reads of hot data are bottlenecked by CPU/RAM): if I have saturated five machines with writes I should have five machines worth of read throughput, and that is a /lot/ of throughput... with the behavior in this article, you have no read capacity as you are blocked by writes, even if the writes are to unrelated collections of data (which is downright non-sensical behavior). Even with more machines (I guess, twice as many), the latency of your reads is now being effected horribly by having to wait for the write locks... it is just a bad solution.

Re: A Year with MongoDB

#19

Post by mitchellh »

zzzeek wrote: post #17
mitchellh wrote: post #7

Earlier quoted context omitted.

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark sin…

typo in the numbers ? I can get your mongoDB number but not pg: >>> seconds_per_month = 60 * 60 * 24 * 30 >>> ops_per_month_pg = 1000 * seconds_per_month >>> ops_per_month_mg = 200 * seconds_per_month >>> 330.0 / ops_per_month_pg * 100 1.2731481481481482e-05 >>> 330.0 / ops_per_month_mg * 100 6.36574074074074e-05

You're absolutely right. Decimal points are hard. Fixed my comment (and noted it).

Re: A Year with MongoDB

#20

Post by fizx »

sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

There are many serialization formats (e.g. Apache Thrift) with versionable schemas. You can do a poor man's Thrift with json, mongo, etc.

It's a common thing to throw thrift or protobuf values in giant key-value stores (cassandra, replicated memcache, hbase). You don't need to migrate unless you change the key format. If you want do on the fly migrations (mostly to save space, and have the ability to delete ancient code paths), you can do them a row at a time. And yes, we do occasionally write a script to loop through the database and migrate all the documents.

Post reply on HN