Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So, that was a bummer

blog.foursquare.com

Post reply on HN
Print view View original

51–60 of 93 posts

Re: So, that was a bummer

#51

Post by shoesfullofdust »

harryh wrote: post #37
dotBen wrote: post #23

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be…

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

The technical details were great. And I'm sure other devs can learn from your predicament. But, as a user, you didn't answer the question, "When can I use this again?"

It broke. You fixed it. But, "Can I expect this to work again?" "Reliably?" All I heard was that it was broken.

It sounded a lot like, "something broke, it took a long time to fix."

Re: So, that was a bummer

#52

Post by dotBen »

harryh wrote: post #37
dotBen wrote: post #23

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be…

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

The down voting on my previous reply is sad but I probably shouldn't be too surprised. The problem of being way too technical for a mainstream audience is a problem many people on Hacker News seem to have, and so no wonder many would disagree with me.

It's silly for someone on Hacker News to say "well I thought the level of detail was fine" - of course you would, like the rest of us you're a technical geek. The point that seems to be lost is 95% of FourSquare's userbase ISN'T!

Also FourSquare is one of those startups that, in addition to the YC startups (for obvious reasons I guess), people give a little more favoritism to then perhaps other startups of equal quality/interestingness.

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

Well, I'm not suggesting you wrote "something broke, it took a long time to fix" - I'm all for transparency. But if you are going to be transparent you need to communicate at a level at which that transparency can be understood by all of your readers. I'm sorry if some people on Hacker News don't get that.

So ok, here's how I would have written your post (for time sake I just did the intro - I'd have repeated the technical description after this block of copy):

Yesterday, we experienced a very long downtime. All told, we were down for about 11 hours, which is unacceptably long. It sucked for everyone (including our team – we all check in everyday, too). We know how frustrating this was for all of you because many of you told us how much you’ve come to rely on foursquare when you’re out and about. For the 32 of us working here, that’s quite humbling. We’re really sorry.

Below is an explanation of what happened and what we’re doing to make sure it doesn’t happen again in the future (a more technical explanation for those inclined appears further below)

What happened As you can imagine we store a huge amount of data from all of your user check-ins. We split that data across many servers as it's obviously far to big to fit onto just one. Starting around 11:00am EST yesterday we noticed that one of these servers was performing poorly because it was receiving an unusually high volume of check-ins. Maybe there was an incredibly popular party that we missed out on! :)

Anyway, after trying various things to improve the performance of that server we decided to try to add another server to take some of the strain off the original overloaded server. We wanted to move this data in the background while the site remained up - however for some reason when we added the new server the entire site did go down. Ouch!

We tried all sorts of things to ease the strain but nothing seemed to work. By around 6:30pm EST (phew, what a day!) we decided to try one final idea, which fortunately worked. Yay!

However it took a further 5 hours to properlly test our fix, and so it was only by around 11:30pm EST that we were able to bring the site back up. Don't worry, all of your data remained safe at all times, and that hard-won mayorship is still yours!

...

Anyway, if people disagree that you should always communicate with your customer at a level they understand, then I'd urge you to read http://steveblank.com/2010/04/22/turning-on-your-reality-dis... or http://www.readwriteweb.com/start/2010/05/is-your-startup-to... (pitching to investors, media or customers - it's all the same issues).

Re: So, that was a bummer

#53

Post by pvg »

sabat wrote: post #12

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not bla…

in 1999, OLTP was still new, and, hence the bug.

No, OLTP was not at all new in 1999. OLTP probably means something other than what you think it means.

Re: So, that was a bummer

#54

Post by mscarborough »

harryh wrote: post #37
dotBen wrote: post #23

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be…

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

I liked that your post didn't start with "we use MongoDB and there's some problems with it". I haven't used MongoDB on production, and while I'm interested to learn about specific issues, the format of this post gives the reader the opportunity to evaluate your problem from a platform-agnostic perspective first. Instead of having Mongo interfere right away, I can think about how our systems might hit the same issues.

To echo others, I'm interested to read the more in-depth post-mortem.

Good luck!

Re: So, that was a bummer

#55

Post by jbooth »

VladRussian wrote: post #34
dmytton wrote: post #7

Earlier quoted context omitted.

Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a sha…

"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results." an approach to CAP that chooses consistency over availability. I wonder why as it isn't a bank and they could have chosen the availability instead.

If in doubt, choose consistency.

Re: So, that was a bummer

#57

Post by rbranson »

Devilboy wrote: post #42
rbranson wrote: post #35

Earlier quoted context omitted.

Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what…

The whole point of MongoDB is that you SHOULDN'T have to worry about a server becoming overloaded! You're giving up a lot for this privilege too, so if that doesn't even work properly then... back to PostgreSQL in my opinion.

MongoDB is designed to help ease the pain of scaling, but what you're asking for is magic. If all the sudden a large amount of requests start coming through that overload a shard (think very popular users, like if several very popular users ended up colocated on a shard), how is MongoDB going to anticipate this? Any kind of shard scaling will only work well in scenarios where you have a reasonable increase in load, not a crippling surge.

In addition, it's very, VERY difficult to scale writes against a single object, such as Justin Bieber's profile data, say, if you've got a view counter on it. You can either serialize writes on read like Cassandra does, which has it's own drawbacks (the more writers an object has, the more expensive reads become), or you can have single-master-for-an-object sharding like MongoDB employs and most other production sites (Facebook, Flickr, etc) use.

Re: So, that was a bummer

#58

Post by jrockway »

mcfunley wrote: post #46
jrockway wrote: post #45

Earlier quoted context omitted.

Wait, people shard on a key other than something approximately random, like an sha1 hash!?

Where I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment. Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).

Sounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases.

I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.

Re: So, that was a bummer

#59

Post by Devilboy »

rbranson wrote: post #57
Devilboy wrote: post #42

Earlier quoted context omitted.

The whole point of MongoDB is that you SHOULDN'T have to worry about a server becoming overloaded! You're giving up a lot for this privilege too, so if that doesn't even work properly then... back to PostgreSQL in my opinion.

MongoDB is designed to help ease the pain of scaling, but what you're asking for is magic. If all the sudden a large amount of requests start coming through that overload a shard (think very popular users, like if several very popular users ended up colocated on a shard), how is MongoDB going to anticipate this? Any kind of shard scaling will only work well in scenarios where you have a reasonable increase in load, n…

There was no sudden crippling surge in this case as far as I can tell. There was no mass-updating of a single object either. It failed even though it fits your ideal situation pretty much perfectly.

Re: So, that was a bummer

#60

Post by roc »

dotBen wrote: post #52
harryh wrote: post #37

Earlier quoted context omitted.

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

The down voting on my previous reply is sad but I probably shouldn't be too surprised. The problem of being way too technical for a mainstream audience is a problem many people on Hacker News seem to have, and so no wonder many would disagree with me. It's silly for someone on Hacker News to say "well I thought the level of detail was fine" - of course you would, like the rest of us you're a technical geek. The point…

While I agree on the technical-level bit, the tone in your sample floats between cavalier and condescending. That's worse than technical overkill IMO.
Post reply on HN