Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So, that was a bummer

blog.foursquare.com

Post reply on HN
Print view View original

31–40 of 93 posts

Re: So, that was a bummer

#31

Post by lipnitsk »

marclove wrote: post #25
dotBen wrote: post #23

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be…

How many "regular users" do you really think would spend the time to read about why they were down? They just know it was down and wanted it back up ASAP. And how would you better explain it to a layman? The servers crashed. We worked on it. It took longer than we wanted. They're back up again and we're working to make sure it doesn't happen again. How do you go further than that without getting technical?

Actually, I think that it wasn't even all that technical. They took their time to explain what shards are, etc, so their intended audience was still power users, at best. Not saying it's good or bad, just saying how I felt when I read the post.

Re: So, that was a bummer

#32

Post by blantonl »

sabat wrote: post #12

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not bla…

I'm unable to come up with any reference to a 3 day outage regarding Oracle and Ebay in 1999. Can you provide more info on this - I'm very interested to see what happened.

Edit: I found this reference to the 22hr outage that occurred, and I remember this outage, but I don't ever remember it being a 3 days outage.

http://www.internetnews.com/ec-news/article.php/137251/Cost-...

Re: So, that was a bummer

#33

Post by dmytton »

haberman wrote: post #27

Having to manually shard your data seems like so much work when offerings like App Engine will take care of that for you. It seems like exactly the kind of thing that you shouldn't have to think about when you're trying to get a business off the ground. I can see the lock-in concerns with AppEngine, but an AppEngine level of abstraction seems so much more appropriate than manually deploying/configuring an entire infr…

It's not manual sharding. You specify a key a MongoDB is supposed to handle everything for you. There are manual operations you can perform if you want (like moving data or splitting) but normally you'd let Mongo handle it all.

Re: So, that was a bummer

#34

Post by VladRussian »

dmytton wrote: post #7
brown9-2 wrote: post #2

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…

Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a sha…

"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results."

an approach to CAP that chooses consistency over availability. I wonder why as it isn't a bank and they could have chosen the availability instead.

Re: So, that was a bummer

#35

Post by rbranson »

marclove wrote: post #4

Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not h…

Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what actually brought the site down completely, hopefully they'll be able to trace it down and fill us in on that.

Re: So, that was a bummer

#36

Post by fizx »

houseabsolute wrote: post #5
brown9-2 wrote: post #2

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…

I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.

Just because the feature exists, doesn't mean it works.

Re: So, that was a bummer

#37

Post by harryh »

dotBen wrote: post #23

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be…

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix."

Would love to hear suggestions on this topic.

Re: So, that was a bummer

#38

Post by mcfunley »

houseabsolute wrote: post #5
brown9-2 wrote: post #2

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go dow…

I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.

I would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.

Re: So, that was a bummer

#40

Post by bhiggins »

If there's anyone from Foursquare here, I'm interested in what monitoring you have in place. How long had the shard been poorly performing before you noticed? Do you use anything to monitor MongoDB in particular, or load on servers, anything like that?
Post reply on HN