Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

41–50 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#41

Post by jasonwatkinspdx »

Building sharded systems isn't as simple as throwing consistent hashing into the code and calling it a day. You have to think carefully about what happens when nodes exceed capacity. There is good work in academia on distribution algorithms that gracefully handle reaching capacity (along with data center structure such as rack awareness) [1]. Alternately, if your algorithm doesn't handle a shard reaching capacity you need to have the monitoring and processes in place to ensure you always add more capacity and rebalance before running out.

Also, this validates Redis's VM position about 4KB pages being to large to properly manage data swapping in web application storage.

[1]: http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf

Re: MongoDB's lead developer: Foursquare outage post mortem

#42

Post by harryh »

fleitz wrote: post #40
harryh wrote: post #33

Earlier quoted context omitted.

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

Why not keep the algorithm and process in batches? You don't need to have EVERY users checkin history in RAM at any moment. Throw the checkin on a queue, have a few processes that query a db for the checkin history and you're fine. Heck, if you delay the awards it's also a good excuse to throw the user an alert to come back to the site/app.

That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.

Re: MongoDB's lead developer: Foursquare outage post mortem

#43

Post by jbarham »

zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM?

If you can fit your db into RAM that's the ideal, but of course you also have to have a reliable, persistent backup. The best current compromise is probably PCI based SSD drives.

John Ousterhout (of Tcl fame) and his group at Stanford are working on a project called RAMCloud that's exploring the feasibility of storing data in RAM at all times, using disk only as backup. See his projects page http://www.stanford.edu/~ouster/cgi-bin/projects.php for more info. IMO, given the continuous improvements in RAM prices and network latencies, this type of setup for permanent storage will be the norm in data centers in a few years.

Re: MongoDB's lead developer: Foursquare outage post mortem

#44

Post by doty »

seiji wrote: post #35
voxxit wrote: post #27

Why they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.

They are dealing with web scale sharded NoSQL realtime geo scala. Old rules don't apply when 80% of the words describing your company didn't exist two years ago.

I honestly cannot tell whether or not that comment was serious.

Re: MongoDB's lead developer: Foursquare outage post mortem

#46

Post by SkyMarshal »

slantyyz wrote: post #25
redthrowaway wrote: post #19

The main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about t…

Forget about Apple, it would have been nice if BP followed this example.

Problem is lawsuits. Admitting guilt can and is used against you in court. Hence, US businesses don't apologize for anything. This is a refreshing exception.

Re: MongoDB's lead developer: Foursquare outage post mortem

#47

Post by IgorPartola »

harryh wrote: post #33
zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

You could set up a user-place relationship and simply increment the number of check-ins for that place, then periodically correct the number, subtracting old ones which no longer count. Or look at the entire history only if there is a chance of getting a badge based on this number and otherwise just increment it. For badges for the total number of check-ins this is even easier, since no recalculation is necessary.

I suppose if your CPU load is not too high you could even run background processes to monitor "near-badge" status for all users with high enough frequency. This you could really get wild with: people with most number of recent check-ins should get scanned first and of course this list should be updated as a separate entity so it stays small and always in memory.

Re: MongoDB's lead developer: Foursquare outage post mortem

#48

Post by jasonwatkinspdx »

jranck wrote: post #4

For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.

When you're a small company growing fast you have an endless list of things you really should have done a long time ago. That's no excuse for not preventing a predictable and painful failure, but do try to remember it's a lot easier to criticize as an observer with no time commitments.

Re: MongoDB's lead developer: Foursquare outage post mortem

#49

Post by harryh »

donaldc wrote: post #28

It sounds like even if the check-ins had grown evenly across the two shards, that would have only saved them for about another month. Without close monitoring of the growth in memory usage, foursquare was still due for an outage.

This is 100% true. We were always planning on moving to more shards, we just thought we had more time to do it than we did.

Re: MongoDB's lead developer: Foursquare outage post mortem

#50

Post by harryh »

moondowner wrote: post #24

One thing that I admire about Foursquare is that they don't only use the technology (MongoDB, Lift Framework, etc..) but also invest in it. If it wasn't for them I think that Lift wouldn't be such an advanced framework as it is now.

Thanks! We do what we can (we're very busy these days!), and only hope to be able to do more of this sort of thing in the future.
Post reply on HN