Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

151–160 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#151

Post by dirkgadsden »

look_lookatme wrote: post #125
dirkgadsden wrote: post #121

Earlier quoted context omitted.

The duration of the downtime seems to have been dependent on not just disk throughput, but also network and CPU throughput. Eliot did mention that a large portion of the downtime was partly caused by the slowness of EBS. Making a rough estimate, I think the downtime would probably be about 2/3 of how long it was if it had been on an SSD RAID. However, then you run into the issue of having to maintain your own servers…

It's an interesting angle if that's the reason they are using EC2. Dedicated hosting might be more expensive, but hardware managed is pretty efficient at this point. Also, given the vast number of EC2 instances they could afford, it seems counter intuitive to be running only two mongrel shards. If you are that stingy about spreading data around, you might as well be using dedicated.

It doesn't seem that it was necessarily stinginess of neglect, but rather stinginess imposed by their situation. Like the article said, it took hours to create a new shard, downtime that Foursquare definitely did not want.

Re: MongoDB's lead developer: Foursquare outage post mortem

#153

Post by marclove »

mrkurt wrote: post #87
voxxit wrote: post #27

Why they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.

They actually said 4. Two shards with slaves for redundancy. It was apparently working fine until they lost track of what was going on. :)

Yea but like you said, since half of them are just for redundancy, they were in effect only running 2 database servers. I must say I find it stunning that a service with that much traffic, especially one with an infrastructure that requires the entire database to be in-memory, would be operating with just 2 database servers running on EC2.

I know hindsight is 20/20, but I can't imagine the foresight was any worse than 20/30.

Re: MongoDB's lead developer: Foursquare outage post mortem

#154

Post by look_lookatme »

look_lookatme wrote: post #125
dirkgadsden wrote: post #121

Earlier quoted context omitted.

The duration of the downtime seems to have been dependent on not just disk throughput, but also network and CPU throughput. Eliot did mention that a large portion of the downtime was partly caused by the slowness of EBS. Making a rough estimate, I think the downtime would probably be about 2/3 of how long it was if it had been on an SSD RAID. However, then you run into the issue of having to maintain your own servers…

It's an interesting angle if that's the reason they are using EC2. Dedicated hosting might be more expensive, but hardware managed is pretty efficient at this point. Also, given the vast number of EC2 instances they could afford, it seems counter intuitive to be running only two mongrel shards. If you are that stingy about spreading data around, you might as well be using dedicated.

Whoops, I meant "mongo" shard!

Re: MongoDB's lead developer: Foursquare outage post mortem

#155

Post by piramida »

So some people still use built-in memory allocation for large-scale memory-intense projects like mongodb eh? Pity that, they should have looked at why exactly memcached (and any other sane piece of memory-write-erase intensive application) does slab allocations.

Re: MongoDB's lead developer: Foursquare outage post mortem

#156

Post by robryan »

This may be a little naive seeing as I haven't used MongoDB or know anything about foursquare architecture, but wouldn't a better result have been just to drop requests that were hitting the disk? Sure some people would have been getting a sub optimal experience but arguably it's a lot better than not having your service up at all.

Re: MongoDB's lead developer: Foursquare outage post mortem

#157

Post by sofuture »

hartror wrote: post #124
there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

2 cents an hour!

Re: MongoDB's lead developer: Foursquare outage post mortem

#158

Post by sofuture »

robryan wrote: post #156

This may be a little naive seeing as I haven't used MongoDB or know anything about foursquare architecture, but wouldn't a better result have been just to drop requests that were hitting the disk? Sure some people would have been getting a sub optimal experience but arguably it's a lot better than not having your service up at all.

From my understanding, that 'routing' decision would have to take place inside of Mongo -- null routing DDoS packets on a network is one thing, dropping certain database requests on the fly inside a database server sounds a littttle bit tricker.

Re: MongoDB's lead developer: Foursquare outage post mortem

#159

Post by cies »

tmountain wrote: post #63

The "give a crap" factor at 10gen is nothing short of amazing. The company I work at has been using Mongo for a while, and anytime we've had an issue, Eliot has been right there to help us. Mongo is a great product, but like anything, it has its limits. Learning those limits and taking the time to plan your infrastructure is a mandatory part of adopting any technology, and Mongo is no exception. On a more technical n…

> Mongo is a minimum two server product to begin with

are you sure? i thought it runs well on one server and even in some sort of run-locally setup.

Re: MongoDB's lead developer: Foursquare outage post mortem

#160

Post by kvs »

Looks like there needs to be some data engineering to be done to back memory up with disk storage. I think you guys, by now, have enough information about users and checkins to create algorithms that can determine "hot" checkins vs. "cold" checkins as and when they come in. Because not every user is the same, a more user-centric view can definitely help you come up with a more scalable (or graciously failing) system. For example, I don't checkin as much as some of my friends who checkin 10-15 times a day. So, there is no point in giving my checkins as much priority as those busy folks.
Post reply on HN