Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So, that was a bummer

blog.foursquare.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: So, that was a bummer

#61

Post by harryh »

dotBen wrote: post #52
harryh wrote: post #37

Earlier quoted context omitted.

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

The down voting on my previous reply is sad but I probably shouldn't be too surprised. The problem of being way too technical for a mainstream audience is a problem many people on Hacker News seem to have, and so no wonder many would disagree with me. It's silly for someone on Hacker News to say "well I thought the level of detail was fine" - of course you would, like the rest of us you're a technical geek. The point…

Interesting. Thx very much fo the feedback. I'm sure you understand it's a hard balance to strike between technical detail and ease of understanding. Will strive to make things a bit more on the "ease of understanding" side next time.

Also considering starting a separate engineering blog where it would probably be appropriate to go into more detail for those that are interested.

-harryh

Re: So, that was a bummer

#62

Post by ora600 »

rbranson wrote: post #35
marclove wrote: post #4

Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not h…

Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what…

Here's what I do in these situations (I'm an Oracle DBA, but this should apply to most loaded shards):

1) Use connection pooling at the application layer to prevent overloading the DB of any specific shard. This means that if a shard has 16 CPUs, having 16 connections sounds reasonable. Additional connections will not give you more performance. This means you need to queue and throttle requests at the application layer and with some thought you can probably figure out what to do with the waiting users - show partial results? show a nice whale? A "loading please wait" sign?

2) If you didn't do #1 and the DB is getting overloaded, my normal response is to start shooting down connections. Oracle has separate unix process per connection. MySQL has its own way of shooting connections down. Put up a small script that will kill the correct percentage of sessions to prevent overload on shared resources. This will generates lots of errors and will cause a percentage of the users to hate you, but you won't be down.

Re: So, that was a bummer

#63

Post by steveklabnik »

marclove wrote: post #15
dmytton wrote: post #10

Earlier quoted context omitted.

You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.

Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?

I'm sure that if you're the size of Foursquare, you can afford the contract.

Then again, it's no excuse for bad software. I've only used Mongo on small sites so far, and have been loving it.

Re: So, that was a bummer

#64

Post by dotBen »

roc wrote: post #60
dotBen wrote: post #52

Earlier quoted context omitted.

The down voting on my previous reply is sad but I probably shouldn't be too surprised. The problem of being way too technical for a mainstream audience is a problem many people on Hacker News seem to have, and so no wonder many would disagree with me. It's silly for someone on Hacker News to say "well I thought the level of detail was fine" - of course you would, like the rest of us you're a technical geek. The point…

While I agree on the technical-level bit, the tone in your sample floats between cavalier and condescending. That's worse than technical overkill IMO.

It would be interesting to know which bits you felt that was?

Also keep in mind I tried to edit their original post, as kinda suggested. I'm not sure I'd have written any the post quite in the way that they did - but I tried to work with what I had.

Re: So, that was a bummer

#65

Post by ankimal »

Firstly, as a free user you really shouldn't be asking for HA. (I m assuming their paid customer are kicking some butt as I speak .. or maybe not .. the site is back up :)).

However, as a business you really want to give ALL your customers HA. Its not just a reputation thing, its a "we love you all equally" kinda attitude.

As for MongoDB, we ve been using in production for small insignificant things. FWIW, they have replication http://www.mongodb.org/display/DOCS/Replication and some cool new features like Replica Sets for failover and redundancy. Maybe they missed a trick?

I think the apology post was totally fair and he did categorically mention ".. This blog post is a bit technical. It has the details of what happened, and what we’re doing to make sure it doesn’t happen again in the future.". They could have dilly-dallied with words and said "we had a technical failure of a data nature" and that would have been just been plain stupid. So thanks for the detailed technical write up and hope there is more to follow.

Re: So, that was a bummer

#66

Post by fizx »

jrockway wrote: post #58
mcfunley wrote: post #46

Earlier quoted context omitted.

Where I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment. Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).

Sounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases. I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.

Even/odd isn't the end of the world, but you would then be best jumping to mod 4.

Re: So, that was a bummer

#67

Post by fizx »

marclove wrote: post #15
dmytton wrote: post #10

Earlier quoted context omitted.

You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.

Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?

I'm sure it was diagnosed quickly. Sometimes you have to copy data, rebuild indexes, etc.

Re: So, that was a bummer

#68

Post by rbranson »

ora600 wrote: post #62
rbranson wrote: post #35

Earlier quoted context omitted.

Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what…

Here's what I do in these situations (I'm an Oracle DBA, but this should apply to most loaded shards): 1) Use connection pooling at the application layer to prevent overloading the DB of any specific shard. This means that if a shard has 16 CPUs, having 16 connections sounds reasonable. Additional connections will not give you more performance. This means you need to queue and throttle requests at the application lay…

1) Application connection pooling won't scale. In a scenario like FourSquare, there is likely a 4:1 ratio of app server to DB shard server. Further, connections don't necessarily equal load.

2) This sounds like a great way to create data inconsistencies, unless you've got very tight constraints on your database, which is impossible in a sharded scenario.

I agree though, that ultimately they should have had some way to "fail whale" instead of getting overloaded.

Re: So, that was a bummer

#69

Post by rb2k_ »

brown9-2 wrote: post #2

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…

> To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?

MongoDB still has problems with locking too much for certain operations. The 10gen guys keep on introducing "yields" to those operations, but the simple (and fast) database/process-level locking could very well cause this... (not a 100% 'yes', but it seems possible)

Re: So, that was a bummer

#70

Post by rb2k_ »

staunch wrote: post #39
houseabsolute wrote: post #9

Earlier quoted context omitted.

http://labs.google.com/papers/bigtable.html

That looks potentially useful in some cases. Can you send me the code/docs for that?

Also look at dynamo based systems like cassandra and riak (riak seems to have a better load balancing at the moment, cassandra is a bit more "bumpy")
Post reply on HN