Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So, that was a bummer

blog.foursquare.com

Post reply on HN
Print view View original

11–20 of 93 posts

Re: So, that was a bummer

#11

Post by marclove »

dmytton wrote: post #7
brown9-2 wrote: post #2

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…

Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a sha…

But it wasn't the overloaded shard that brought the site down (it was only slowing things down), it was the addition of the new shard:

"For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down."

Re: So, that was a bummer

#12

Post by sabat »

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug.

I'm not blaming a flat-out bug in this case (the cause of the severe part is still unknown?), but it could also be architectural or operator error.

Re: So, that was a bummer

#13

Post by smoody »

What I don't understand is this: Why do companies (like four square, twitter, etc) wait until after their first multi-hour crash before instituting a "this is how we'll communicate to users when we have downtime" process? I would assume that everyone has learned from twitter's historical mistakes at this point. I would argue that startups -- especially startups that deal with large numbers of transactions per day -- start with code and policies for communicating downtime issues first and launch the product second.

Re: So, that was a bummer

#14

Post by dmytton »

marclove wrote: post #11
dmytton wrote: post #7

Earlier quoted context omitted.

Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a sha…

But it wasn't the overloaded shard that brought the site down (it was only slowing things down), it was the addition of the new shard: "For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down."

If the data was being moved it could cause even more load on an already overloaded shard.

Re: So, that was a bummer

#15

Post by marclove »

dmytton wrote: post #10
marclove wrote: post #4

Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not h…

You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.

Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this.

But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?

Re: So, that was a bummer

#16

Post by dmytton »

houseabsolute wrote: post #5
brown9-2 wrote: post #2

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…

I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.

Load based splitting is on the MongoDB roadmap, but doesn't exist yet.

Re: So, that was a bummer

#17

Post by ddlatham »

smoody wrote: post #13

What I don't understand is this: Why do companies (like four square, twitter, etc) wait until after their first multi-hour crash before instituting a "this is how we'll communicate to users when we have downtime" process? I would assume that everyone has learned from twitter's historical mistakes at this point. I would argue that startups -- especially startups that deal with large numbers of transactions per day --…

It's not part of a minimum viable product. When you're trying to get something out the door, your downtime communication is not your top priority, so you can afford to improve it later.

Re: So, that was a bummer

#18

Post by mickeyben »

smoody wrote: post #13

What I don't understand is this: Why do companies (like four square, twitter, etc) wait until after their first multi-hour crash before instituting a "this is how we'll communicate to users when we have downtime" process? I would assume that everyone has learned from twitter's historical mistakes at this point. I would argue that startups -- especially startups that deal with large numbers of transactions per day --…

By experience, I'd say it was on their roadmap, there was even maybe a ticket about having a status.foursquare.com.

But you know there's tons of tickets, tons of priorities and finally shit happens and some tasks are placed on top of the pile and become priorities.

Re: So, that was a bummer

#19

Post by japherwocky »

ddlatham wrote: post #17
smoody wrote: post #13

What I don't understand is this: Why do companies (like four square, twitter, etc) wait until after their first multi-hour crash before instituting a "this is how we'll communicate to users when we have downtime" process? I would assume that everyone has learned from twitter's historical mistakes at this point. I would argue that startups -- especially startups that deal with large numbers of transactions per day --…

It's not part of a minimum viable product. When you're trying to get something out the door, your downtime communication is not your top priority, so you can afford to improve it later.

aka "cross that bridge when you come to it"

Re: So, that was a bummer

#20

Post by Sukotto »

Well, that's another example for the "dangers of making untested changes to your production environment" pile. Of course, people rarely feel the need to post when everything works out...

Sucks that such a popular service had such trouble. I look forward to reading any additional posts they write explaining in more detail exactly what happened.

Post reply on HN