Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So, that was a bummer

blog.foursquare.com

Post reply on HN
Print view View original

41–50 of 93 posts

Re: So, that was a bummer

#41

Post by thegoleffect »

harryh wrote: post #37
dotBen wrote: post #23

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be…

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

Having more MongoDB-specific technical info might be a call-to-arms for other MongoDBAs who might offer their ideas/thoughts. You could also use that as a hiring/scouting opportunity.

Does 4sq have an engineering blog?

Re: So, that was a bummer

#42

Post by Devilboy »

rbranson wrote: post #35
marclove wrote: post #4

Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not h…

Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what…

The whole point of MongoDB is that you SHOULDN'T have to worry about a server becoming overloaded! You're giving up a lot for this privilege too, so if that doesn't even work properly then... back to PostgreSQL in my opinion.

Re: So, that was a bummer

#44

Post by amock »

sabat wrote: post #12

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not bla…

Are you sure that was due to an Oracle bug? I heard of one eBay outage while watching a talk on ZFS that was due to misconfigured SAN volumes that overlapped and caused repeated data corruption in the Oracle DB, but wasn't related to any bug in Oracle.

Re: So, that was a bummer

#45

Post by jrockway »

mcfunley wrote: post #38
houseabsolute wrote: post #5

Earlier quoted context omitted.

I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.

I would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.

Wait, people shard on a key other than something approximately random, like an sha1 hash!?

Re: So, that was a bummer

#46

Post by mcfunley »

jrockway wrote: post #45
mcfunley wrote: post #38

Earlier quoted context omitted.

I would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.

Wait, people shard on a key other than something approximately random, like an sha1 hash!?

Where I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment.

Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).

Re: So, that was a bummer

#47

Post by jackowayed »

waxman wrote: post #21

The site just went down again, awkwardly enough, only moments after they published their post-mortem on yesterday's outage. Clearly, as their blog post indicates, they were unable to trace the root problem. To me, the worst feeling in the world as a developer is when there's a major bug in your production site, and you can't figure out exactly why it happened. Then even after you get the site working there's that pit…

Not surprisingly, it is a related problem:

> UPDATE Oct. 5 8:01PM: Our server team is still working to resolve the problem. The issue is related to yesterday’s outage.

Re: So, that was a bummer

#48

Post by gonehome »

harryh wrote: post #37
dotBen wrote: post #23

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be…

How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.

I liked your blog post and appreciated the level of detail you put into it. I'm always wondering why things went down and what's going on behind the scenes and I'm sure many other users appreciate it too. Those that don't likely won't mind the extra information.

Re: So, that was a bummer

#49

Post by cagenut »

I notice foursquare.com resolves to ec2, was the mongodb that got overloaded also on ec2? Can you tell us in what way did it get "overloaded" (iowait, mem/swap, raw-cpu)?
Post reply on HN