Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

161–170 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#161

Post by tmountain »

cies wrote: post #159
tmountain wrote: post #63

The "give a crap" factor at 10gen is nothing short of amazing. The company I work at has been using Mongo for a while, and anytime we've had an issue, Eliot has been right there to help us. Mongo is a great product, but like anything, it has its limits. Learning those limits and taking the time to plan your infrastructure is a mandatory part of adopting any technology, and Mongo is no exception. On a more technical n…

> Mongo is a minimum two server product to begin with are you sure? i thought it runs well on one server and even in some sort of run-locally setup.

Yes, if you care about your data, it's a two server product. Single server redundancy isn't being added until 1.8. Until then, if the server terminates abnormally, there's potential for data corruption. The solution right now is to run a slave.

http://www.mongodb.org/display/DOCS/Durability+and+Repair

Re: MongoDB's lead developer: Foursquare outage post mortem

#162

Post by parfe »

megablast wrote: post #73
redthrowaway wrote: post #19

The main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about t…

What are you talking about? How can you see how the users see foursquare after this? Sure, in the hacker community this detail is appreciated, but how do you know how many foursquare users have left because of these outages? How many people are upset with the company, and are now taking facebook locations or the other companies more seriously?

Left over this? Foursquare isn't a bank. Foursquare is a game. An achievement system for work lunches.

Of the people I know who use four square the response was "Hm, foursquare isn't working" puts phone away.

Re: MongoDB's lead developer: Foursquare outage post mortem

#163

Post by jhugg »

Devilboy wrote: post #117
there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

Please take the time to capitalize your sentences, your otherwise great comment deserves it.

You really should have considered a semi-colon over a comma. ;-)

Re: MongoDB's lead developer: Foursquare outage post mortem

#164

Post by sausagefeet »

tmountain wrote: post #161
cies wrote: post #159

Earlier quoted context omitted.

> Mongo is a minimum two server product to begin with are you sure? i thought it runs well on one server and even in some sort of run-locally setup.

Yes, if you care about your data, it's a two server product. Single server redundancy isn't being added until 1.8. Until then, if the server terminates abnormally, there's potential for data corruption. The solution right now is to run a slave. http://www.mongodb.org/display/DOCS/Durability+and+Repair

This solution strikes me as best-case-scenario error. What happens if the connection between my slave and master goes down then my master is unplugged? What happens if both slave and master are unplugged, can both databases be corrupted? What am I missing?

Re: MongoDB's lead developer: Foursquare outage post mortem

#165

Post by 83457 »

hartror wrote: post #124
there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

Do they have server admin on staff, or is the whole point of hosting in the cloud to not need them?

Re: MongoDB's lead developer: Foursquare outage post mortem

#166

Post by [deleted] »

trezor wrote: post #145
rbranson wrote: post #98

Earlier quoted context omitted.

Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of pe…

I have had very acceptable performance for latency critical applications where the db have exceeded the available memory by a factor between 10 and 50. So far I have never heard of any one running a commercial RDBMS reiterate the MySQL-mantra that you need the entire DB in RAM and I find it a very puzzling attitude to efficient database usage.

[deleted]

Re: MongoDB's lead developer: Foursquare outage post mortem

#167

Post by metageek »

hartror wrote: post #142
binspace wrote: post #135

Earlier quoted context omitted.

Other things get prioritized.

Like not having a massive downtime that gives their users a reason to try out some massive social network's offering that has already started eating their lunch? Oh wait no they didn't prioritize that. My point is, and perhaps the down vote indicates I didn't make it well, that monitoring and alerts are extremely important for any startup. They admit as much in the article, had they known sooner there wouldn't have b…

Now where did I put the key to that barn door?

It's in the saddlebags.

Re: MongoDB's lead developer: Foursquare outage post mortem

#168

Post by piyush_ranjan »

harryh wrote: post #33
zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

NO

Re: MongoDB's lead developer: Foursquare outage post mortem

#169

Post by binspace »

hartror wrote: post #142
binspace wrote: post #135

Earlier quoted context omitted.

Other things get prioritized.

Like not having a massive downtime that gives their users a reason to try out some massive social network's offering that has already started eating their lunch? Oh wait no they didn't prioritize that. My point is, and perhaps the down vote indicates I didn't make it well, that monitoring and alerts are extremely important for any startup. They admit as much in the article, had they known sooner there wouldn't have b…

Humans don't judge probability and risk very well. It's hard to forsee the future and problems. Hindsight is 20/20. My bag of cliches is exhausted.

Maybe there were other things the engineers were working on that brought definite (as in 100%) benefit. Maybe there's a good product/consulting opportunity here. Everybody needs monitoring, but it takes time and resources.

I agree, that they are focusing on it now, because it happened.

Re: MongoDB's lead developer: Foursquare outage post mortem

#170

Post by tmountain »

sausagefeet wrote: post #164
tmountain wrote: post #161

Earlier quoted context omitted.

Yes, if you care about your data, it's a two server product. Single server redundancy isn't being added until 1.8. Until then, if the server terminates abnormally, there's potential for data corruption. The solution right now is to run a slave. http://www.mongodb.org/display/DOCS/Durability+and+Repair

This solution strikes me as best-case-scenario error. What happens if the connection between my slave and master goes down then my master is unplugged? What happens if both slave and master are unplugged, can both databases be corrupted? What am I missing?

If the connection goes away and your master is unplugged, you will need to run a repair on the master and then bring the slave back into sync. If both are unplugged, you will have to run the repair on both. This obviously isn't ideal, and that's probably why single server durability is the biggest priority for the next release.
Post reply on HN