Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

21–30 of 153 posts

Re: A Year with MongoDB

#21

Post by jabwork »

mitchellh wrote: post #7
jrussbowman wrote: post #3

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scalin…

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark sin…

Can you give any sort of indication of the value of a schemaless database and the flexibility it provided as the team fleshed out the data model? Was this a mere convenience over traditional schema migration or something more?

Re: A Year with MongoDB

#23

Post by geoffeg »

madworld wrote: post #16
paulsutter wrote: post #4

This article is an excellent articulation of the strengths and (fixable) issues with mongoDB. I like MongoDB a lot, and the improvements suggested would really strengthen the product and could make me more comfortable to use it in more serious applications.

How is the global write lock "fixable" without a major rewrite of the codebase? Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.

DB-level locking is planned for MongoDB 2.2 which should be out within a few months.

https://jira.mongodb.org/browse/SERVER-4328

Re: A Year with MongoDB

#24

Post by bitops »

sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"

I think that is arguably one of the selling points of MongoDB. Yes, you do implement schema in your application, but should be doing that knowingly and embracing both the costs and benefits.

The benefit is that you can very quickly change your "schema" since it's just however you're choosing to represent data in memory (through your objects or whatever). It also allows you to have part of your application running on one "version" of the schema while another part of the application catches up.

The tradeoff is that you have to manage all this yourself. MongoDB does not know about your schema, nor does it want to, nor should it. It affords you a lot of power, but then you have to use it responsibly and understand what safety nets are not present (which you may be used to from a traditional RDBMS).

To your point about migrations, there are a few different strategies. You can do a "big bang" migration where you change all documents at once. (I would argue that if you require all documents to be consistent at all times, you should not be using MongoDB). A more "Mongo" approach is to migrate data as you need to; e.g. you pull in an older document and at that time add any missing fields.

So yes, in MongoDB, the schema effectively lives in your application. But that is by design and to use MongoDB effectively that's something you have to embrace.

Re: A Year with MongoDB

#25

Post by radicalbyte »

sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> oh, so you're implementing schema in your application?

Isn't that where the schema belongs? Each document represents a conceptual whole. It doesn't contain fields which have to be NULL simply because they weren't in previous versions of the schema.

I've been an rdbms guy (datawarehousing/ETL) for a long time now, I've seen a lot of large databases which have been in production for considerable time. They get messy. Really messy. They become basically unmaintainable. Apples, oranges and pears all squashed into a schema the shape of a banana.

It's a pretty elegant solution, and is the problem XML/XSD were designed to solve.

The cleanest solution that I've seen in production used a relational database as a blob storage for XML-serialized entities. Each table defined a basic interface for the models, but each model was free to use its own general schema. After 10 years it contained a set of very clean individual entities which were conceptually correct.

As opposed to the usage as a serialization format for remoting, which has been largely replaced with JSON.

Re: A Year with MongoDB

#26

Post by justinsb »

antirez wrote: post #11
LeafStorm wrote: post #5

Earlier quoted context omitted.

While I agree that you should definitely use your tools in the best way you're capable of, I think for most people there's a baseline expectation that if you save data in a database, that data will be safe (at the very least, recoverable) unless something happens like the server catching on fire. Nearly every other major database has this as the default -- MySQL, PostgreSQL, CouchDB, and Berkeley DB to name a few. (R…

You are wrong about Redis. Cheers

Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.

Re: A Year with MongoDB

#27

Post by luca_garulli »

Hey, by reading all the bad things seems that OrientDB would fit better than MongoDB for them:

- Non-counting B-Trees: OrientDB uses MVRB-Tree that has the counter. size() requires 0ns

- Poor Memory Management: OrientDB uses MMAP too but with many settings to optimize it usage

- Uncompressed field names: the same as OrientDB

- Global write lock: this kills your concurrency! OrientDB handles read/write locks at segment level so it's really multi-thread under the hood

- Safe off by default: the same as OrientDB (turn on synch to stay safe or use good HW/multiple servers)

- Offline table compaction: OrientDB compacts at each update/delete so the underlying segments are always well defragmented

- Secondaries do not keep hot data in RAM: totally different because OrientDB is multi-master

Furthermore you have Transactions, SQL and support for Graphs. Maybe they could avoid to use a RDBMS for some tasks using OrientDB for all.

My 0,02.

Re: A Year with MongoDB

#28

Post by spidaman »

geoffeg wrote: post #23
madworld wrote: post #16

Earlier quoted context omitted.

How is the global write lock "fixable" without a major rewrite of the codebase? Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.

DB-level locking is planned for MongoDB 2.2 which should be out within a few months. https://jira.mongodb.org/browse/SERVER-4328

Meh, if your other option is to use PostgreSQL and get row level locks, a db level lock is still a fail.

Re: A Year with MongoDB

#29

Post by [deleted] »

luca_garulli wrote: post #27

Hey, by reading all the bad things seems that OrientDB would fit better than MongoDB for them: - Non-counting B-Trees: OrientDB uses MVRB-Tree that has the counter. size() requires 0ns - Poor Memory Management: OrientDB uses MMAP too but with many settings to optimize it usage - Uncompressed field names: the same as OrientDB - Global write lock: this kills your concurrency! OrientDB handles read/write locks at segmen…

[deleted]

Re: A Year with MongoDB

#30

Post by fusiongyro »

mitchellh wrote: post #7
jrussbowman wrote: post #3

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scalin…

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark sin…

I like your post, and I agree with your conclusion, but I have to say I'm puzzled by your decision to back MongoDB with EBS. Were you running MongoDB atop EC2 instances as well? Can you elaborate on this a little?
Post reply on HN