MongoDB's lead developer: Foursquare outage post mortem

Post by heresy » Fri, Oct 08, 2010, 6:06 AM UTC

hartror wrote: post #124

there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

Post by NIGGERFUCKER » Fri, Oct 08, 2010, 6:11 AM UTC

jemfinch wrote: post #118

there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

> how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? The high-memory quadruple extra large (no, I'm not making that name up) instance offers 68.4GB of ram. Presumably they left the rest for the OS.

Post by trezor » Fri, Oct 08, 2010, 6:39 AM UTC

rbranson wrote: post #98

jasonwatkinspdx wrote: post #34

Earlier quoted context omitted.

Facebook runs primarily out of ram via memcached. The last numbers I'm aware of were that they had about 200TB in memcache capacity [1]. They use a variety of data stores, but primarily sharded mysql. I don't have recent numbers there, but they were above 1000 master-master pairs as of 2008. While buying that much ram sound costly, that's only looking at capacity. Assuming typical 1u servers and common pricing at the…

Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of pe…

Post by [deleted] » Fri, Oct 08, 2010, 6:45 AM UTC

t_crayford wrote: post #67

jasonwatkinspdx wrote: post #41

Building sharded systems isn't as simple as throwing consistent hashing into the code and calling it a day. You have to think carefully about what happens when nodes exceed capacity. There is good work in academia on distribution algorithms that gracefully handle reaching capacity (along with data center structure such as rack awareness) [1]. Alternately, if your algorithm doesn't handle a shard reaching capacity you…

I'm wondering if the decision to shard based on users was taken with data (ie, at that point did each user have a roughly similar number of check-ins); if not, hashing by that seems kind of fail for that kind of app.

Post by mfukar » Fri, Oct 08, 2010, 9:56 AM UTC

jasonwatkinspdx wrote: post #119

rcoder wrote: post #90

Earlier quoted context omitted.

From the email thread, it sounds like the decision to shard on UID was made mostly to increase locality of data, so that you didn't have to query more than one node to get a single user's data. There's no silver bullet here. Hashing on insertion order would basically guarantee that writes would favor one node over another, which random hashes would force you to aggregate results from all available nodes for each quer…

This stuff can be very counter intuitive. Locality may not be what you want. For example, last I heard google's search index was sharded by document rather than by term. That sounds odd, since if it was sharded by term, then a given search would only need to go to a handful of servers (one for each term) and then the intermediate result combined. But with it sharded by document, every query has to go to all the nodes…