Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So, that was a bummer

blog.foursquare.com

Post reply on HN
Print view View original

71–80 of 93 posts

Re: So, that was a bummer

#71

Post by moe »

fizx wrote: post #66
jrockway wrote: post #58

Earlier quoted context omitted.

Sounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases. I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.

Even/odd isn't the end of the world, but you would then be best jumping to mod 4.

Actually for anonymous sharding (without a central index) a consistent hash is about the closest you can get to ideal distribution and flexibility. I haven't looked but I presume that's what mongo uses under the hood for their auto-sharding, too.

Re: So, that was a bummer

#72

Post by marclove »

steveklabnik wrote: post #63
marclove wrote: post #15

Earlier quoted context omitted.

Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?

I'm sure that if you're the size of Foursquare, you can afford the contract. Then again, it's no excuse for bad software. I've only used Mongo on small sites so far, and have been loving it.

Its all fun and games until someone gets a shard in the eye.

Re: So, that was a bummer

#74

Post by fauigerzigerk »

dmytton wrote: post #7
brown9-2 wrote: post #2

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…

Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a sha…

I know nothing about MongoDB but why would one shard that's responding slowly affect queries to all the other shards and take the whole site down? I can't believe any database system would be designed to work like that.

Re: So, that was a bummer

#75

Post by fauigerzigerk »

fizx wrote: post #67
marclove wrote: post #15

Earlier quoted context omitted.

Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?

I'm sure it was diagnosed quickly. Sometimes you have to copy data, rebuild indexes, etc.

According to that blog post it hasn't actually been properly diagnosed yet. They see the symptoms but they don't know why they're seeing those symptoms.

Re: So, that was a bummer

#76

Post by fauigerzigerk »

rbranson wrote: post #57
Devilboy wrote: post #42

Earlier quoted context omitted.

The whole point of MongoDB is that you SHOULDN'T have to worry about a server becoming overloaded! You're giving up a lot for this privilege too, so if that doesn't even work properly then... back to PostgreSQL in my opinion.

MongoDB is designed to help ease the pain of scaling, but what you're asking for is magic. If all the sudden a large amount of requests start coming through that overload a shard (think very popular users, like if several very popular users ended up colocated on a shard), how is MongoDB going to anticipate this? Any kind of shard scaling will only work well in scenarios where you have a reasonable increase in load, n…

How is this primarily an issue of overlaod? One particular shard got overlaoded, yes, but the reall issue is how that brought the whole system down.

Re: So, that was a bummer

#77

Post by fauigerzigerk »

dotBen wrote: post #52
harryh wrote: post #37

Earlier quoted context omitted.

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

The down voting on my previous reply is sad but I probably shouldn't be too surprised. The problem of being way too technical for a mainstream audience is a problem many people on Hacker News seem to have, and so no wonder many would disagree with me. It's silly for someone on Hacker News to say "well I thought the level of detail was fine" - of course you would, like the rest of us you're a technical geek. The point…

I disagree (but I never downvote). Leaving out technical detail, translating everything into end user friendly but meaningless phraseology is the worst thing you can do. Microsoft has done that in all its consumer products and it's infuriating because users have no way of asking someone more knowledgable for help.

I think end users have a problem if the one thing they need to know is expressed in a way they cannot understand. What they need to know is when the site is going to be up again and what the likelihood of it happening again is. Once they know that, I don't think they have a problem with added technical detail that's not meant for everyone.

Re: So, that was a bummer

#78

Post by jscore »

I'm so happy they're back up, here I was thinking the world would come to a screeching halt when people cannot check-in to places.

Seriously though, are they THAT important?

Re: So, that was a bummer

#79

Post by cyu »

marclove wrote: post #15
dmytton wrote: post #10

Earlier quoted context omitted.

You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.

Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?

If I was 10Gen, it would be in my best interest to offer support to 4sq, contract or not. People (like us) are watching.

Re: So, that was a bummer

#80

Post by cyu »

haberman wrote: post #27

Having to manually shard your data seems like so much work when offerings like App Engine will take care of that for you. It seems like exactly the kind of thing that you shouldn't have to think about when you're trying to get a business off the ground. I can see the lock-in concerns with AppEngine, but an AppEngine level of abstraction seems so much more appropriate than manually deploying/configuring an entire infr…

I can't imagine any co with that much investment money betting on a platform that they can never take ownership of.
Post reply on HN