So, that was a bummer
blog.foursquare.com
So, that was a bummer
1–10 of 93 posts
Re: So, that was a bummer
#2Interestingly the cause of the most severe part of their downtime seems to still be unknown to them:
As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one.
We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down.
To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?
Re: So, that was a bummer
#3we noticed that one of these shards was performing poorly because a disproportionate share of check-ins were being written to it.
I'd love to know the root cause behind this specific issue. Was this a behavioral issue within the user base, or a technical problem that routed check-ins to this specific shard more than others?
Since they mentioned they partition their shards by userId that would probably rule out their routing process. I wonder if there was some event that caused a certain sharded subsection of users to start sending so many checkins?
And since this was a subset of userIds on the same shard - could this have been a targeted DOS or SPAM event?
I'm making a very conscious migration to MongoDB so I'm very interested to hear what the root cause of this was.
Re: So, that was a bummer
#4Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not having someone on staff who really understands all the ins & outs of a database technology their entire service depends upon.
Re: So, that was a bummer
#5Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…
I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
Re: So, that was a bummer
#6Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…
I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
Which NoSQL store do you use?
Re: So, that was a bummer
#7Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…
Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available.
But it sounds like the shard was still up. As per the sharding FAQ[1]:
"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results."
They said they had a performance issue on the overloaded shard so perhaps it's possible that mongo believed the shard to be up, when it was instead overloaded. This meant any queries just waited, taking down the whole site.
As I said, this is speculation and as a massive production user of MongoDB ourselves[2], I'm interested to know more.
[1] http://www.mongodb.org/display/DOCS/Sharding+FAQ [2] http://blog.boxedice.com/2010/02/28/notes-from-a-production-...
Re: So, that was a bummer
#8Ironically, the servers are being "upgraded" again, as we speak.
Re: So, that was a bummer
#9Earlier quoted context omitted.
I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
Which NoSQL store do you use?
Re: So, that was a bummer
#10Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not h…
You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.