Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling MongoDB at Mailbox

tech.dropbox.com

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Scaling MongoDB at Mailbox

#71

Post by idProQuo »

misiti3780 wrote: post #45
rdtsc wrote: post #42

Earlier quoted context omitted.

Marketing. They marketed the hell out of it. Most developers I know have MongoDB mug. > Lately I haven't read any news about a company migrating to Mongo, but rather most were either departing from mongo They had some strange defaults to start with (to give them serious advantage in small silly benchmarks) like un-acknowledged writes. Yes you read that correctly, for years their default configuration was to throw wri…

or - mongo is great for some problems and not so great for other problems. if there is one thing you can figure out hanging out on HN long enough - it's cool to hate on mongodb

I want to like Mongo, I'm all about "cutting my teeth on the bleeding edge", but I don't know what to think about MongoDB. Most conversations about it seem to go like this:

Lover: MongoDB is so much faster and easier than old school SQL! Hater: It's fast and easy because it doesn't check if it's written data correctly, which leads to corrupt data. Lover: Well you could always turn on the write lock. Hater: If you turn on the write lock, it becomes slower than normal SQL databases. Lover: Well you're just trying to use it the same way you use SQL. MongoDB is great for some problems and not so great for other problems. Hater: What problems is MongoDB better at solving?

And then that's the end. I've never heard a convincing use case for this thing. If you can change that for me and give me one, I'd be delighted to listen.

Re: Scaling MongoDB at Mailbox

#72

Post by annnnd »

justinsb wrote: post #11
lucisferre wrote: post #4

Earlier quoted context omitted.

The problems people encounter with SQL performance generally (or at least also) are on the read side where query optimization can become a pretty hard problem. Basically, the short answer is people are using SQL and noSQL very differently (as one would expect). cue: someone telling me SQL query optimization is easy and people are just idiots.

It's a valid point taken by itself. But I would respectfully suggest that it's not a particularly useful statement, because you're not comparing SQL to anything. "Complex SQL queries are hard" because "complex queries are hard". If there was a magical tool where you could do complex querying and always have the right answer in the minimal time, then these discussions would never arise. (To a debatable extent, that is…

The caveat being that in document-oriented store (like MongoDB) the queries tend to be simple(r) - after all, you have no joins, so how complex can they be? Instead of complex queries you need to use appropriate document schemes and/or application logic.

MongoDB has its problems (for instance document schemes can't be enforced, disk space consumption concept is unintuitive,...) and relational DBs have their strengths (ACID, SQL is standard among DBs,...). But very few people ever reach a point where the global write lock becomes a problem. 10gen knows exactly which features are vital to most of their users and which are not...

Re: Scaling MongoDB at Mailbox

#73

Post by lucian1900 »

annnnd wrote: post #70
lucian1900 wrote: post #67

Earlier quoted context omitted.

Only with unacknowledged writes. If you ask for acks, it gets ridiculously slow. Much, much slower in every single way than Postgres with a fully indexed HSTORE. That's also fully ACID.

Of course Postgres is also relational. Have you tried using a document oriented store like MongoDB? I have. For some problems it just plain rocks. That doesn't mean I would want to store financial transactions for a bank in a MongoDB, but it has its place and that place is clearly huge (judging by the number of people using it). I love it because it is: - simple to use - fast enough (with safe writes, thank you!) - s…

Postgres' HSTORE is as document oriented as Mongo.

I have used Mongo plenty and I really don't want to do it again. With its almost-safe writes it's extremely slow and no simpler to use than Postgres. Also much less flexible, basically a subset of Postgres' data model.

Re: Scaling MongoDB at Mailbox

#74

Post by taude »

stonewhite wrote: post #18

I dont understand how MongoDB can stay as a shiny tool given its long list of shortcomings. It is so elastic that I think it really has no true form (object cache, mq, relational??, key-value bucket). Lately I haven't read any news about a company migrating to Mongo, but rather most were either departing from mongo because of some catastrophic outage or some mission impossible operation to handle the shortcomings, li…

I think the premise of prototyping tool is pretty accurate. It makes easy to get something up and running and defer any operational knowledge until later. That being said, I don't see why people can't iterate their apps quickly on time-proven relational datastores. While having JavaScript through the stack is nice, I found writing Mongo Queries a PITA with all the curly-braces, etc...

Re: Scaling MongoDB at Mailbox

#75

Post by justinsb »

annnnd wrote: post #72
justinsb wrote: post #11

Earlier quoted context omitted.

It's a valid point taken by itself. But I would respectfully suggest that it's not a particularly useful statement, because you're not comparing SQL to anything. "Complex SQL queries are hard" because "complex queries are hard". If there was a magical tool where you could do complex querying and always have the right answer in the minimal time, then these discussions would never arise. (To a debatable extent, that is…

The caveat being that in document-oriented store (like MongoDB) the queries tend to be simple(r) - after all, you have no joins, so how complex can they be? Instead of complex queries you need to use appropriate document schemes and/or application logic. MongoDB has its problems (for instance document schemes can't be enforced, disk space consumption concept is unintuitive,...) and relational DBs have their strengths…

But you still need to join in NoSQL, for any query that isn't along the natural hierarchy of the data.

The relational model takes this observation and concludes that there should be no "natural hierarchy" in the logical model (the physical model is a separate question.) It's a _theoretically_ beautiful idea. The counter-intuitive _real-world_ result is that the theoretical approach also yielded faster systems than alternative philosophies. I think that's the reason why the relational model has dominated for the past 30+ years.

You might well argue that that's because a _good_ NoSQL implementation has yet to be created (the no true Scotsman argument). That may well prove to be true one day, but I would bet that a good non-relational implementation will be a derivative from the relational world (Postgres with hstore, or Google's F1 with Protobufs); not from a project whose starting axiom is to get rid of SQL/relational.

Re: Scaling MongoDB at Mailbox

#76

Post by justinsb »

Diederich wrote: post #58
stonewhite wrote: post #53

Earlier quoted context omitted.

One thing you don't know or faced yet is that mongo fails, and sometimes it fails so hard that you would rather kick some rocks. For example, if you start a background index generation on an eventually consistent replica set, indexing on secondary nodes are done foreground. Which means you only accept reads from slaves but slaves are unresponsive because of the index generation. In this state, if you try to do anythi…

I've been running, on a five minute cron, kill -9 on the master mongo instance in our QA lab for a good long time. 24/7. There's a program running that, 100 times per second, reads a document, MD5's the field, and writes it back. At the same time, it reads a file from the local filesystem, MD5's it, and writes it back. The document and the local filesystem file started with the same value. After a few thousand kill -…

Is your test producing a new document value 100 times a second, or just writing the same value back over and over again?

It sounds like it might be the latter, which is not a particularly stressful test (because you can't detect data rollback).

I'm more familiar with relational database internals, but I wouldn't be surprised if a DB just optimized out the unchanged-write entirely (they'd still need to read the current row value, but they don't have to invoke any data modification code once they see the value hasn't changed).

For a good test, you really want to simulate a power-loss, which you aren't getting when you do a process-level kill, because all the OS cache/buffers survives. You can do simulate this with a VM, or with a loopback device. I'd be amazed if MongoDB passed a changing-100-times-a-second test then. I'd be amazed if any database passed it. I'd even be amazed if two filesystems passed :-)

Re: Scaling MongoDB at Mailbox

#77

Post by lucian1900 »

tkellogg wrote: post #40
lucian1900 wrote: post #27

Earlier quoted context omitted.

Yes, MongoDB really is that bad. Never use it.

Use rethinkdb instead. Its not perfect but it mainly hits the same niche but does it right.

Or anything else, for that matter. Postgres, MySQL, even SQLite beats Mongo for any use-case. Cassandra, HBase if you need a distributed DB. RethinkDB when it gets mature.

MongoDB is in its own class of terribleness.

Re: Scaling MongoDB at Mailbox

#78

Post by Diederich »

justinsb wrote: post #76
Diederich wrote: post #58

Earlier quoted context omitted.

I've been running, on a five minute cron, kill -9 on the master mongo instance in our QA lab for a good long time. 24/7. There's a program running that, 100 times per second, reads a document, MD5's the field, and writes it back. At the same time, it reads a file from the local filesystem, MD5's it, and writes it back. The document and the local filesystem file started with the same value. After a few thousand kill -…

Is your test producing a new document value 100 times a second, or just writing the same value back over and over again? It sounds like it might be the latter, which is not a particularly stressful test (because you can't detect data rollback). I'm more familiar with relational database internals, but I wouldn't be surprised if a DB just optimized out the unchanged-write entirely (they'd still need to read the curren…

I'm reading the document, hashing it, and writing the hashed value back. So it changes every time.

I plan on extending this test by blocking 27017 with iptables, then doing the kill -9, then wiping out all of the database files. That'll be fun. :)

Post reply on HN