MongoDB's lead developer: Foursquare outage post mortem • Hacker News Board

Re: MongoDB's lead developer: Foursquare outage post mortem

#111

Post by Nate75Sanders » Fri, Oct 08, 2010, 2:06 AM UTC

wheels wrote: post #72

donaldc wrote: post #31

In essence, although we had moved 5% of the data from shard0 to the new third shard, the data files, in their fragmented state, still needed the same amount of RAM. This can be explained by the fact that Foursquare check-in documents are small (around 300 bytes each), so many of them can fit on a 4KB page. Removing 5% of these just made each page a little more sparse, rather than removing pages altogether. Interestin…

I don't think those are the same problem -- could you provide a link? The problem isn't paging, per se -- the paging system is doing exactly what it should be doing, paging in blocks off of the disk and into memory as they become hot, which for this use case is always. The problem is that you get fragmentation in your pages. If you allocate three records in a row that are 300 bytes, and then need to rewrite the first…

It seems that if you know your page size and you know your record size, you can compute the optimal size into which you should break your records into chunks and link the chunks with pointers. This causes more memory accesses, but surely that's worth keeping everything in memory? More chunks per record means more pointers and more accesses, but it should still be way faster in many cases.

How to compare this to compaction, in the general case, isn't immediately obvious to me.

Re: MongoDB's lead developer: Foursquare outage post mortem

#112

Post by there » Fri, Oct 08, 2010, 2:10 AM UTC

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server.

i'm baffled at how there could be no monitoring or reporting in place to catch that days or weeks ahead of time, let alone just 12 hours needed according to the mongodb developer, to fix the problem without downtime. it's such a fundamental thing to keep track of for a system designed entirely around storing a bunch of data in the memory of those 2(!) machines. i have more servers than foursquare and none of them do even a small fraction of the amount of processing that theirs do, and yet i have real-time bandwidth, memory, cpu, and other stats being collected, logged, and displayed, as well as nightly jobs that email me various pieces of information. nobody at foursquare ever even logged into those systems and periodically checked the memory usage manually?

worse still, during all of this, the initial outage reports were blaming mongodb or saying the problem was unknown. even at that point nobody at foursquare realized that the servers were just out of memory?

how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? was there some kind of capacity planning to come up with that number or is it just a hard limit of EC2 and the foursquare developers maxed out the configuration?

Re: MongoDB's lead developer: Foursquare outage post mortem

#113

Post by tav » Fri, Oct 08, 2010, 2:12 AM UTC

rit wrote: post #17

narrator wrote: post #10

Well Foursquare's Scala/Lift front-end sure has been holding up nicely despite all the FUD about Lift's stateful architecture. Funny that Mongo DB, which gets all the scalability hype, is the first thing to have trouble scaling.

The described problem - imbalance in shard allocation - is not one that's specific to MongoDB. In fact one of the huge choices one has to make when deploying say, Cassandra, is the partitioning algorithm. Sequential partitioning (saying say, users a-l go on Partition 1 and m-z go on #2) gives you certain capabilities for range queries but also risks you overloading if one particular user is significantly larger than…

Or your datastore can take care of doing that for you, c.f. BigTable. This is one of my main gripes with most of the current set of NoSQL offerings — they leave too many decisions in the hands of developers.

Whilst it would definitely be advantageous for all developers to understand the intricacies of various CPU and OS scheduling algorithms, it's not an issue that most developers have to deal with directly. The App Engine datastore, in particular, proves that it is possible to create NoSQL datastores which don't force developers to think about issues like load balancing.

Re: MongoDB's lead developer: Foursquare outage post mortem

#114

Post by look_lookatme » Fri, Oct 08, 2010, 2:12 AM UTC

While I understand this is foremost a monitoring and architecture (mongo and 4sq side...) issue, but I'm curious how dangerous it is to run such IO dependent systems on EC2. Would the prolonged downtime (due to shard migration, etc) been as severe if they were running on hardware? What if these two servers were on SSD RAID 0?

Re: MongoDB's lead developer: Foursquare outage post mortem

#115

Post by rbranson » Fri, Oct 08, 2010, 2:15 AM UTC

harryh wrote: post #110

rbranson wrote: post #105

Earlier quoted context omitted.

Have you thought about how much extra overhead there is in storing the checkins using MongoDB's BSON structures? Considering how often you do this, you might consider packing a specialized version of a user's history (at least for this purpose -- sort of a denormalization) into a array of 64-bit placeIDs? This means a checkin would only occupy 8 bytes of space and there would be no overhead. 1 billion checkins = 8GB…

Unfortunately we have some badges that require more than just the venueid + timestamp, but some sort of "compressed" checkin format is something we'll probably end up looking at.

Bummer. It's really amazing how 1 seemingly simple (but quite important and engagement driving) feature like instant gratification badges can drive your architecture decisions and keep you guys up at night.

Re: MongoDB's lead developer: Foursquare outage post mortem

#116

Post by brown9-2 » Fri, Oct 08, 2010, 2:20 AM UTC

jmulho wrote: post #106

ww520 wrote: post #94

Earlier quoted context omitted.

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the…

I was curious about a similar question to yours: if they had 66GB of RAM, why did going just slightly over the threshold cause such drastic paging for them - surely queries aren't touching all parts of their dataset equally?

harryh's answer above about querying the user's entire history for each checkin answers this question though: http://news.ycombinator.com/item?id=1769909

Re: MongoDB's lead developer: Foursquare outage post mortem

#117

Post by Devilboy » Fri, Oct 08, 2010, 2:23 AM UTC

there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

Please take the time to capitalize your sentences, your otherwise great comment deserves it.

Re: MongoDB's lead developer: Foursquare outage post mortem

#118

Post by jemfinch » Fri, Oct 08, 2010, 2:23 AM UTC

there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

> how did the developers come up with 66 gigabytes of ram to use for these instances in the first place?

The high-memory quadruple extra large (no, I'm not making that name up) instance offers 68.4GB of ram. Presumably they left the rest for the OS.

Re: MongoDB's lead developer: Foursquare outage post mortem

#119

Post by jasonwatkinspdx » Fri, Oct 08, 2010, 2:25 AM UTC

rcoder wrote: post #90

t_crayford wrote: post #67

Earlier quoted context omitted.

I'm wondering if the decision to shard based on users was taken with data (ie, at that point did each user have a roughly similar number of check-ins); if not, hashing by that seems kind of fail for that kind of app.

From the email thread, it sounds like the decision to shard on UID was made mostly to increase locality of data, so that you didn't have to query more than one node to get a single user's data. There's no silver bullet here. Hashing on insertion order would basically guarantee that writes would favor one node over another, which random hashes would force you to aggregate results from all available nodes for each quer…

This stuff can be very counter intuitive. Locality may not be what you want.

For example, last I heard google's search index was sharded by document rather than by term.

That sounds odd, since if it was sharded by term, then a given search would only need to go to a handful of servers (one for each term) and then the intermediate result combined. But with it sharded by document, every query has to go to all the nodes in each replica/cluster.

It ends up that's not as bad as it seems. Since everything is in ram, they can answer "no matches" on a given node extremely quickly (using bloom filters mostly in on processor cache or the like). They also only send back truly matching results, rather than intermediates that might be discarded later, saving cluster bandwidth. Lastly it means their search processing is independent of the cluster communication, giving them a lot of flexibility to tweak the code without structural changes to their network, etc.

Does that mean everyone doing search should shard the same way? Probably not. You have to design this stuff carefully and mind the details. Using any given data store is not a silver bullet.

Re: MongoDB's lead developer: Foursquare outage post mortem

#120

Post by jmulho » Fri, Oct 08, 2010, 2:32 AM UTC

Can we clear up one issue? Sharding on userid -- bad or fine? I say fine. If your algorithm is "if userid between A and M then server0, else (N-Z) server1", then you could get unbalanced. But if your algorithm is "if random 50/50 hash of userid = 0 then server0, else (random 50/50 hash = 1) server1", then you are going to stay almost exactly balanced (law of large numbers). You would never, ever, get anywhere near a 60/40 imbalance. So the choice to use userid, if implemented correctly, is completely fine.