MongoDB's lead developer: Foursquare outage post mortem
groups.google.com
MongoDB's lead developer: Foursquare outage post mortem
1–10 of 184 posts
Re: MongoDB's lead developer: Foursquare outage post mortem
#2117 GB of data and 300 B for a checkin means that Foursquare has had 408,944,640 checkins.
Re: MongoDB's lead developer: Foursquare outage post mortem
#3117 GB of data and 300 B for a checkin means that Foursquare has had 408,944,640 checkins.
There are numerous indexes on the data - so actual data is about half of that.
Re: MongoDB's lead developer: Foursquare outage post mortem
#4For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves.
Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.
Re: MongoDB's lead developer: Foursquare outage post mortem
#5That was one of the most well-written, to-the-point explanations I have read in a long time.
Re: MongoDB's lead developer: Foursquare outage post mortem
#6117 GB of data and 300 B for a checkin means that Foursquare has had 408,944,640 checkins.
They hit 200 million on Oct 6. http://techcrunch.com/2010/10/06/foursquare-200-million/
Re: MongoDB's lead developer: Foursquare outage post mortem
#7For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.
A bit harsh but a good point. There are a few red flags here. Resource monitoring on the servers, like you mentioned, seems pretty obvious. Especially considering that the system was almost guaranteed to crash when it ran out of RAM.
Sharding needs to be managed in a way that evenly distributes the data. I would never do it by user ID unless a proper analysis of the data showed it to be a fairly even distribution.
Re: MongoDB's lead developer: Foursquare outage post mortem
#8I wonder if there ought to be a move to allow for sharded systems to lump groups/pages nicely.. kind of like MySQL's partitioning.. where you won't run into the issue of sparsing out pages when you migrate data.. instead you'd effectively migrate pages.
Re: MongoDB's lead developer: Foursquare outage post mortem
#9I wonder if there ought to be a move to allow for sharded systems to lump groups/pages nicely.. kind of like MySQL's partitioning.. where you won't run into the issue of sparsing out pages when you migrate data.. instead you'd effectively migrate pages.
Yes - something that keeps data organized my shard key is being planned.
Re: MongoDB's lead developer: Foursquare outage post mortem
#10Well Foursquare's Scala/Lift front-end sure has been holding up nicely despite all the FUD about Lift's stateful architecture. Funny that Mongo DB, which gets all the scalability hype, is the first thing to have trouble scaling.