Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

101–110 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#101

Post by bmm6o »

megablast wrote: post #68
donaldc wrote: post #31

In essence, although we had moved 5% of the data from shard0 to the new third shard, the data files, in their fragmented state, still needed the same amount of RAM. This can be explained by the fact that Foursquare check-in documents are small (around 300 bytes each), so many of them can fit on a 4KB page. Removing 5% of these just made each page a little more sparse, rather than removing pages altogether. Interestin…

Surely if they moved more than 5% across, this would have freed up more memory, despite the fragmentation. Maybe they would be better identifying regular users, these would require on the fly compacting, be hosted on one or two machines, with a third smaller server for the non-frequent users.

Freeing memory would require that there be an entirely empty page. Each page holds 13-14 objects (4k/300). The chances that 13 consecutive objects were in the 5% (assuming independent, random distribution) is 1 in 20^13, putting the expected number of empty pages well below 1. You have to migrate much more data before you can hope to see page-size holes.

Re: MongoDB's lead developer: Foursquare outage post mortem

#102

Post by ww520 »

harryh wrote: post #33
zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

This is similar to one of the problem I used to deal with in tracking huge volume of billing events. Since the data (checkins) are immutable, it's fairly easy to build summary running totals daily. The total count then becomes the running-total plus the current day's count. The summary job can be run offline in the background. Just make sure to mark the current day's events as processed and update the running total at the same transaction.

In this way, the amount of current day's events are pretty small and you don't need to keep all historic events in memory.

Re: MongoDB's lead developer: Foursquare outage post mortem

#103

Post by donaldc »

wheels wrote: post #89
donaldc wrote: post #78

Earlier quoted context omitted.

I don't think those are the same problem -- could you provide a link? You are correct, the actual problem is paging out LRU keys as opposed to memory holes. The issue is related but not the same. From http://antirez.com/post/what-is-wrong-with-2006-programming.... Multiply this for all the keys you have in memory and try visualizing it in your mind: These are a lot of small objects. What happens is simple to explain,…

What antirez is getting at there is actually a much harder (and more interesting) problem that could be generalized as something like "efficient data locality for mixed latency access". However, assuming that all data must actually be in memory (as stated in the posted email), you don't actually solve the Mongo problem with more efficient organization of the data set, though compacting could be considered a sub-probl…

I doubt every last bit of their data needs to be hot (i.e. in memory) at any given time, but without specialized paging along the lines of what antirez has discussed for redis, enough of their data probably needs to be hot that, from a paging perspective, all of the vm pages of their data need to be hot.

Re: MongoDB's lead developer: Foursquare outage post mortem

#104

Post by [deleted] »

megablast wrote: post #73
redthrowaway wrote: post #19

The main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about t…

What are you talking about? How can you see how the users see foursquare after this? Sure, in the hacker community this detail is appreciated, but how do you know how many foursquare users have left because of these outages? How many people are upset with the company, and are now taking facebook locations or the other companies more seriously?

[deleted]

Re: MongoDB's lead developer: Foursquare outage post mortem

#105

Post by rbranson »

harryh wrote: post #33
zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

Have you thought about how much extra overhead there is in storing the checkins using MongoDB's BSON structures? Considering how often you do this, you might consider packing a specialized version of a user's history (at least for this purpose -- sort of a denormalization) into a array of 64-bit placeIDs? This means a checkin would only occupy 8 bytes of space and there would be no overhead. 1 billion checkins = 8GB of data. You could even keep these in memcached and use a sort of write-through strategy.

EDIT: I thought about this a bit more, and the checkins are probably time-sensitive, so another 32-bit timestamp (Foursquare Epoch) would need to be stored for each checkin.

Re: MongoDB's lead developer: Foursquare outage post mortem

#106

Post by jmulho »

ww520 wrote: post #94
jmulho wrote: post #91

Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ? Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM. Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recent…

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.

Re: MongoDB's lead developer: Foursquare outage post mortem

#108

Post by look_lookatme »

seiji wrote: post #35
voxxit wrote: post #27

Why they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.

They are dealing with web scale sharded NoSQL realtime geo scala. Old rules don't apply when 80% of the words describing your company didn't exist two years ago.

I cracked up at this. Well played.

Re: MongoDB's lead developer: Foursquare outage post mortem

#109

Post by nkallen »

harryh wrote: post #84
nkallen wrote: post #80

Earlier quoted context omitted.

Can you design the system such that you can turn off ("darkmode") features in an emergency? For example, in an emergency you can turn off badge-awarding as it is known to be RAM intensive (and degrade by seeking to disk). To the extent that you do this across the board, you'll have yet another tool to defend against being over capacity.

We have a framework for doing this sort of thing but it's only implemented around certain features so it doesn't provide us a ton of benefit at the moment. Obviously doing work to give us more control like this could be a big win, so we'll probably be doing some of that.

There is no better time than during an outage to add these features. In my experience, band-aids are better way to resolve an outage than fixing root-causes. Make the problem go away by commenting out code, returning empty result-sets, (or whatever) - and re-balance your partitioned database at some later date.

Re: MongoDB's lead developer: Foursquare outage post mortem

#110

Post by harryh »

rbranson wrote: post #105
harryh wrote: post #33

Earlier quoted context omitted.

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

Have you thought about how much extra overhead there is in storing the checkins using MongoDB's BSON structures? Considering how often you do this, you might consider packing a specialized version of a user's history (at least for this purpose -- sort of a denormalization) into a array of 64-bit placeIDs? This means a checkin would only occupy 8 bytes of space and there would be no overhead. 1 billion checkins = 8GB…

Unfortunately we have some badges that require more than just the venueid + timestamp, but some sort of "compressed" checkin format is something we'll probably end up looking at.
Post reply on HN