Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

181–184 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#181
post #145

Earlier quoted context omitted.

I have had very acceptable performance for latency critical applications where the db have exceeded the available memory by a factor between 10 and 50. So far I have never heard of any one running a commercial RDBMS reiterate the MySQL-mantra that you need the entire DB in RAM and I find it a very puzzling attitude to efficient database usage.

While you may just be trying to talk-down on the MySQL dudes, I don't think that "RAM is faster than disk, so you're more performant if you can fit your data in RAM" is really all that puzzling.

I'm not debating that RAM is faster than disk. I'm debating that you need to have the entire DB in RAM to get good performance.

Most of the time, less than 10% of your DB represents the active working-set of your data and a good database-system should be able to analyse what is being used, gather statistics about the data, use those statistics to intelligently optimize your queries and minimize the need for disk IO.

For any decent RDBMS, having to match total memory with the database size, is wasting money on RAM for very little extra gain in performance and this doesn't really make much business sense.

A concrete example would be a server I manage. It has 16GBs of RAM and handles around 5000 GBs worth of DBs. Due to a good RDBMS and intelligent caching and use of statistics, that server has a cache-hit ratio of 98%. That means that only 2% of queries entering the system results in actual disk IO.

In order to get the last 2% of queries to get into RAM, I would have to increase the system memory by a factor of 30. At that point just getting another server and setting up replication would be much cheaper and also allow a theoretical doubling of troughput.

Really. Having the DB all in RAM is not something which makes marginal sense at all when you compare it to the costs it involves. No amount of internet-argument can convince me that this goal is anything besides a pure waste of money.

Re: MongoDB's lead developer: Foursquare outage post mortem

#182
post #181

Earlier quoted context omitted.

While you may just be trying to talk-down on the MySQL dudes, I don't think that "RAM is faster than disk, so you're more performant if you can fit your data in RAM" is really all that puzzling.

I'm not debating that RAM is faster than disk. I'm debating that you need to have the entire DB in RAM to get good performance. Most of the time, less than 10% of your DB represents the active working-set of your data and a good database-system should be able to analyse what is being used, gather statistics about the data, use those statistics to intelligently optimize your queries and minimize the need for disk IO.…

There's a reason I mentioned object inter-access times. To properly size storage you need to know your query distribution.

"It has 16GBs of RAM and handles around 5000 GBs worth of DBs. Due to a good RDBMS and intelligent caching and use of statistics, that server has a cache-hit ratio of 98%."

This may be your query distribution but it plainly is not everyone's.

"No amount of internet-argument can convince me that this goal is anything besides a pure waste of money."

If Jim Gray's published work doesn't convince you there's not really much left to discuss.

Re: MongoDB's lead developer: Foursquare outage post mortem

#183
post #90

Earlier quoted context omitted.

From the email thread, it sounds like the decision to shard on UID was made mostly to increase locality of data, so that you didn't have to query more than one node to get a single user's data. There's no silver bullet here. Hashing on insertion order would basically guarantee that writes would favor one node over another, which random hashes would force you to aggregate results from all available nodes for each quer…

This stuff can be very counter intuitive. Locality may not be what you want. For example, last I heard google's search index was sharded by document rather than by term. That sounds odd, since if it was sharded by term, then a given search would only need to go to a handful of servers (one for each term) and then the intermediate result combined. But with it sharded by document, every query has to go to all the nodes…

Sharding a search system by documents gives you several advantages. You can scale horizontally by adding additional indexes for new documents. You can tweak a group of documents more easily. E.g. rank wikipedia higher. Assign better hardware (if necessary), higher priority, etc. Performance is also more uniform. Easier to index content at different frequencies. It's also easier for re-indexing content after tweaking algorithms.

If you shard a search index by term instead, you will end up with duplicate documents stored in each index that contain the same term. And for a large index, you need to scale up your hardware to handle the term or shard by document within that term anyway.

Re: MongoDB's lead developer: Foursquare outage post mortem

#184
post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

> how there could be no monitoring or reporting in place

No. off Developers: x No. of Sysadmins: 0

It's a different job, often with a different career path, and a very differnt attitude.

Though some of us get to be be both Code Monkey and the Scary Monk.

Post reply on HN