Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
MongoDB's lead developer: Foursquare outage post mortem
11–20 of 184 posts
Re: MongoDB's lead developer: Foursquare outage post mortem
#12For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.
[deleted]
Re: MongoDB's lead developer: Foursquare outage post mortem
#13Well Foursquare's Scala/Lift front-end sure has been holding up nicely despite all the FUD about Lift's stateful architecture. Funny that Mongo DB, which gets all the scalability hype, is the first thing to have trouble scaling.
MySQL has trouble scaling if you don't understand how to use it or monitor it. Not sure I agree with the last part of your comment at all.
Re: MongoDB's lead developer: Foursquare outage post mortem
#14Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
It's okay to have hundreds of terrabytes of data but you need to keep your "working set" hot. Even if you have 10TB of data, 1TB of it might actually be hot.
Monitoring your working set is tough.
Re: MongoDB's lead developer: Foursquare outage post mortem
#15Well Foursquare's Scala/Lift front-end sure has been holding up nicely despite all the FUD about Lift's stateful architecture. Funny that Mongo DB, which gets all the scalability hype, is the first thing to have trouble scaling.
[deleted]
Re: MongoDB's lead developer: Foursquare outage post mortem
#16I like the way MongoDB describes the flaws in their own system, but places the blame (four) squarely on the customer, where it belongs.
It wasn't a random failure or a sudden spike that caused the crash -- it was completely predictable growth. Foursquare had already experienced the problem once, and they had solved it. All they needed to do was monitor their growth and iterate that solution.
Sure, Foursquare could have had a better sharding algorithm, but that would only have put off the crash a bit longer. This is a very basic failure -- not monitoring a system that you know is steadily growing.
Re: MongoDB's lead developer: Foursquare outage post mortem
#17Well Foursquare's Scala/Lift front-end sure has been holding up nicely despite all the FUD about Lift's stateful architecture. Funny that Mongo DB, which gets all the scalability hype, is the first thing to have trouble scaling.
The described problem - imbalance in shard allocation - is not one that's specific to MongoDB.
In fact one of the huge choices one has to make when deploying say, Cassandra, is the partitioning algorithm. Sequential partitioning (saying say, users a-l go on Partition 1 and m-z go on #2) gives you certain capabilities for range queries but also risks you overloading if one particular user is significantly larger than others.
Cassandra recommends random partitioning as a way of better balancing your data across shards.
You're going to run into the same problem on just about any sharded setup (With any software) - how do you make sure that you are distributing new chunks/documents/rows in a way that doesn't overload any given server.
Re: MongoDB's lead developer: Foursquare outage post mortem
#18For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.
A bit harsh but a good point. There are a few red flags here. Resource monitoring on the servers, like you mentioned, seems pretty obvious. Especially considering that the system was almost guaranteed to crash when it ran out of RAM. Sharding needs to be managed in a way that evenly distributes the data. I would never do it by user ID unless a proper analysis of the data showed it to be a fairly even distribution.
It's probably a technical debt thing. FourSquare is immensely popular, and the team probably meant to have those checks and balances in place but got 'too busy' to implement them.
That doesn't excuse anything, but these oversights can happen even when you've got primo talent on board.
Re: MongoDB's lead developer: Foursquare outage post mortem
#19The main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about the issue, I come away with respect for both groups. Sure, mistakes were made, but they treated their users like adults and took responsibility for what they did wrong.
Good job, guys. If only the likes of Apple followed this example.
Re: MongoDB's lead developer: Foursquare outage post mortem
#20Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
It's okay to have hundreds of terrabytes of data but you need to keep your "working set" hot. Even if you have 10TB of data, 1TB of it might actually be hot. Monitoring your working set is tough.
Yea, but that doesn't look like what they're describing here. They seem to be describing having the entire dataset in memory.