Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling MongoDB at Mailbox

tech.dropbox.com

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Scaling MongoDB at Mailbox

#11

Post by justinsb »

lucisferre wrote: post #4
MichaelGG wrote: post #3

>one performance issue that impacted us was MongoDB’s database-level write lock. People give MSSQL shit for having row-level locks (if you don't use their MVCC option), yet how is it that Mongo runs with a database-wide option and people don't immediately laugh and walk away? Is the hype so powerful that people just shrug about a huge mutex?

The problems people encounter with SQL performance generally (or at least also) are on the read side where query optimization can become a pretty hard problem. Basically, the short answer is people are using SQL and noSQL very differently (as one would expect). cue: someone telling me SQL query optimization is easy and people are just idiots.

It's a valid point taken by itself. But I would respectfully suggest that it's not a particularly useful statement, because you're not comparing SQL to anything. "Complex SQL queries are hard" because "complex queries are hard". If there was a magical tool where you could do complex querying and always have the right answer in the minimal time, then these discussions would never arise. (To a debatable extent, that is one of the things people are paying for when they choose a "better" database: the hope that they get closer to the ideal on complex queries)

Personally, I would much rather perform a complex query in SQL than NoSQL. The base-line is likely to be not-terrible, which may be good enough for single-use queries. Optimization is much easier (IMHO) - SQL is a DSL for querying and indexing. Compared to having to write code to do that (which seems to be the primary NoSQL approach), the DSL approach seems much more efficient. SQL isn't perfect, but it beats no-DSL-support or another DSL that is even worse than SQL :-)

Re: Scaling MongoDB at Mailbox

#12

Post by kareemk »

MichaelGG wrote: post #3

>one performance issue that impacted us was MongoDB’s database-level write lock. People give MSSQL shit for having row-level locks (if you don't use their MVCC option), yet how is it that Mongo runs with a database-wide option and people don't immediately laugh and walk away? Is the hype so powerful that people just shrug about a huge mutex?

We moved to TokuMX (http://www.tokutek.com/products/tokumx-for-mongodb/) recently and saw a 2x improvement in response times. They've swapped out mongo's backend with their fractal tree storage engine which is MVCC. Definitely worth checking out.

Re: Scaling MongoDB at Mailbox

#13

Post by raj_bains »

You need consistency (distributed mvcc), row-level locking and scale-out architecture. Clustrix might be a better fit for the workload. Clustrix customer Twoo.com has 336 core deployment (168 master, 168 slave, 21 nodes each), they have millions of users and billions of transactions per day. Their application still thinks it's talking to a single MySQL database and they don't have a DBA and they have never thought about shard keys etc. We in the database industry should be solving these problems for you.

Re: Scaling MongoDB at Mailbox

#15

Post by spamizbad »

kareemk wrote: post #12
MichaelGG wrote: post #3

>one performance issue that impacted us was MongoDB’s database-level write lock. People give MSSQL shit for having row-level locks (if you don't use their MVCC option), yet how is it that Mongo runs with a database-wide option and people don't immediately laugh and walk away? Is the hype so powerful that people just shrug about a huge mutex?

We moved to TokuMX ( http://www.tokutek.com/products/tokumx-for-mongodb/ ) recently and saw a 2x improvement in response times. They've swapped out mongo's backend with their fractal tree storage engine which is MVCC. Definitely worth checking out.

This is something I've considered pushing for at my company. Any pitfalls you encountered? Did it reduce db size on disk as much as their marketing claims?

Re: Scaling MongoDB at Mailbox

#17

Post by mlmilleratmit »

Nice work -- that's a hard problem. I've felt some of this pain as I helped Cloudant customers make hot migrations from mongo to into Cloudant. Your first two figures make it clear just how challenging hot-replications are (especially master-master), not to mention handling failure scenarios. For all of the great things about Mongo, there's something very awesome to be said for CouchDB's MVCC replication model. Glad to see that you're open sourcing those tools. Maybe they could be extended to make a MongoDB CouchDB replicator.

Re: Scaling MongoDB at Mailbox

#18

Post by stonewhite »

I dont understand how MongoDB can stay as a shiny tool given its long list of shortcomings. It is so elastic that I think it really has no true form (object cache, mq, relational??, key-value bucket). Lately I haven't read any news about a company migrating to Mongo, but rather most were either departing from mongo because of some catastrophic outage or some mission impossible operation to handle the shortcomings, like this one. In its current state it looks more like a prototyping tool that allows you to delay some tech decisions until your product matures, rather than a production tool. Or it is just me that wants at least 5 major versions on his production database.
Post reply on HN