Net impact analysis of outage: A bunch of socially awkward technophiles couldn't tell their friends that they were eating lunch.
MongoDB's lead developer: Foursquare outage post mortem
21–30 of 184 posts
Re: MongoDB's lead developer: Foursquare outage post mortem
#22Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
Depends on what you are doing. If you have high volume and care about latency, you pretty much have to put everything in RAM.
If your disk is spinning at 6000 rpm, a given sector spins past 100 times per second, which makes lookup time anywhere from 0 to 1/100th of a second, or 1/200th of a second on average. Disks only let you look for a limited number of things at once, so you can do less than 1000 disk seeks per second. Period.
If data is living on disk and you have high query volume, it is really, really easy to blow past that limit. The solution is to shard data on multiple machines in RAM. This gives a fixed cost per unit of RAM. As long as you don't exceed available RAM, it works well. Luckily it isn't hard to monitor available RAM and respond in advance. (They didn't do that in this case.)
If you don't care about latency, or have a lower query volume, then you can live with data on disk, and frequently accessed data in RAM. A few well-designed caching layers in front can give you even more headroom. This is much cheaper. But has more complicated failure modes, and they can be tricky to monitor properly.
Re: MongoDB's lead developer: Foursquare outage post mortem
#23Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
It's okay to have hundreds of terrabytes of data but you need to keep your "working set" hot. Even if you have 10TB of data, 1TB of it might actually be hot. Monitoring your working set is tough.
It is easy to turn swap off. There are many reasons why it can be good to turn swap off in servers. (But only if you've got sufficient monitoring to hear about it before you run out of RAM.)
Re: MongoDB's lead developer: Foursquare outage post mortem
#24One thing that I admire about Foursquare is that they don't only use the technology (MongoDB, Lift Framework, etc..) but also invest in it. If it wasn't for them I think that Lift wouldn't be such an advanced framework as it is now.
Re: MongoDB's lead developer: Foursquare outage post mortem
#25The main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about t…
Forget about Apple, it would have been nice if BP followed this example.
Re: MongoDB's lead developer: Foursquare outage post mortem
#26Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
[deleted]
Re: MongoDB's lead developer: Foursquare outage post mortem
#27Why they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.
Re: MongoDB's lead developer: Foursquare outage post mortem
#28It sounds like even if the check-ins had grown evenly across the two shards, that would have only saved them for about another month. Without close monitoring of the growth in memory usage, foursquare was still due for an outage.
Re: MongoDB's lead developer: Foursquare outage post mortem
#29Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
In general you want to have memcached implemented for stuff like this. Have most used stuff closer/faster with memcached.
I don't understand why they are running the whole nosql db on those monster machines - it just defeats the purpose. And sharding architecture that they are mentioning is quite error prone. I would go with read/write seperate channels for things of their nature - it appears they don't have that either.
Re: MongoDB's lead developer: Foursquare outage post mortem
#30Net impact analysis of outage: A bunch of socially awkward technophiles couldn't tell their friends that they were eating lunch.
I wonder if burglaries also went down during the outage, since the bad guys couldn't tell if their targets were at home or busy being the Mayor at the local bowling alley.