MongoDB's lead developer: Foursquare outage post mortem • Hacker News Board

Re: MongoDB's lead developer: Foursquare outage post mortem

#171

Post by lr » Sat, Oct 09, 2010, 7:20 AM UTC

heresy wrote: post #141

hartror wrote: post #124

Earlier quoted context omitted.

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

If it's anything like here, any one engineer's task list is essentially a weighted list of fires to put out and technical debt to pay down, in addition to the features to be delivered next day / week. Guess more extensive monitoring just got bumped to the top :)

Which is exactly why the "don't optimize now" and/or "wait until you have users before you worry about capacity planning" crowd is simply clueless. You need to think about your data and how it will scale from day one!

Re: MongoDB's lead developer: Foursquare outage post mortem

#172

Post by steveklabnik » Sat, Oct 09, 2010, 3:56 PM UTC

trezor wrote: post #145

rbranson wrote: post #98

Earlier quoted context omitted.

Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of pe…

I have had very acceptable performance for latency critical applications where the db have exceeded the available memory by a factor between 10 and 50. So far I have never heard of any one running a commercial RDBMS reiterate the MySQL-mantra that you need the entire DB in RAM and I find it a very puzzling attitude to efficient database usage.

While you may just be trying to talk-down on the MySQL dudes, I don't think that "RAM is faster than disk, so you're more performant if you can fit your data in RAM" is really all that puzzling.

Re: MongoDB's lead developer: Foursquare outage post mortem

#173

Post by steveklabnik » Sat, Oct 09, 2010, 3:59 PM UTC

SriniK wrote: post #29

zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

In general you want to have memcached implemented for stuff like this. Have most used stuff closer/faster with memcached. I don't understand why they are running the whole nosql db on those monster machines - it just defeats the purpose. And sharding architecture that they are mentioning is quite error prone. I would go with read/write seperate channels for things of their nature - it appears they don't have that eit…

Your first three sentences are kind of funny; many people actually consider memecached a NoSQL data store by now.

Re: MongoDB's lead developer: Foursquare outage post mortem

#174

Post by dedward » Sat, Oct 09, 2010, 5:07 PM UTC

83457 wrote: post #165

hartror wrote: post #124

Earlier quoted context omitted.

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

Do they have server admin on staff, or is the whole point of hosting in the cloud to not need them?

Don't know about them - but there seems to be a somewhat scary, hopefully brief, undercurrent out there where business (startups, corporations, whatever) think they can work solely with a few developers and some outsourced services - nobody is really taking responsibility for overall architecture and growth.

(again, NOT saying this applies to foursquare, I have no idea)

Re: MongoDB's lead developer: Foursquare outage post mortem

#175

Post by dedward » Sat, Oct 09, 2010, 5:13 PM UTC

jranck wrote: post #4

For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.

Perhaps it's just my own background that pushes me this way - but as soon as my deployment was growing ot eat up anything over about 60% of my available memory, I'd be looking at deploying more. Your high water mark for spinning up more nodes and/or upgrading shouldn't be your maximum physical ram capacity - that's just nuts.

Re: MongoDB's lead developer: Foursquare outage post mortem

#176

Post by dedward » Sat, Oct 09, 2010, 5:15 PM UTC

zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

It can be entirely acceptable and preferred. Knowing your working set size and how your memory is being utilized is a critical part of any systems planning... and memory is one heck of a lot faster than disk, hands down, no debate there.

Not we're not talking necessarily about a ramdisk here or specialized software (though we could be) - we're talking about proper system design with enough ram and tuning to get the response times you need.

Re: MongoDB's lead developer: Foursquare outage post mortem

#177

Post by dedward » Sat, Oct 09, 2010, 5:18 PM UTC

jbarham wrote: post #43

zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM? If you can fit your db into RAM that's the ideal, but of course you also have to have a reliable, persistent backup. The best current compromise is probably PCI based SSD drives. John Ousterhout (of Tcl fame) and his group at Stanford are working on a project called RAMCloud that's exploring the feasibility of storing data in RAM at all times, using disk on…

It doesn't mean the DB files have to be stored in a ramdisk or something like that - you just need enough ram so your working set can fit in ram.

These new projects really seem a lot like re-hashing problems that have already been solved over and over again - people just keep forgetting to do proper systems engineering in the first place. There is no magic bullet.

Re: MongoDB's lead developer: Foursquare outage post mortem

#178

Post by dedward » Sat, Oct 09, 2010, 5:21 PM UTC

chewbranca wrote: post #60

cullenking wrote: post #59

Earlier quoted context omitted.

Awww come on, it's easy to say from an outside perspective. Regardless, the problem was handled well in the end, and we all get the benefit of understanding these limitations better. I think us tech people got the good side (information) out of this ordeal :)

I work on a team of 2 where I'm responsible for a handful of servers. I chimed in because I'm in a similar position, I've been looking for a monitoring solution for a while now. Things like nagios and zenoss are over kill, but lack of time has prevented me from finding an ideal solution. That said, I keep htop open and running at all times, and its saved my ass on more than one occasion. I say htop because of the col…

Stop thinking about it and set it up. There isn't anything to think about. Nagios (I like using it through groundwork) are certainly not overkill... it's thinking like that that is what gets you in trouble in the first place.

Get that tool in place NOW while you are small, so it's there when you get bigger... so you can hire people to watch it while you move on to other things.

You will always have a lack of time, and always be busy. Spend a weekend putting up Nagios, setting up alerts over SMS or XMPP or Twitter or whatever you want, and then move on. This is fundamental to systems management.

Dont' over think it - just get nagios/cacti (again, try groundwork) up and running and graph EVERYTHING you can... yuo'll be very, very glad you did.

Re: MongoDB's lead developer: Foursquare outage post mortem

#179

Post by dedward » Sat, Oct 09, 2010, 5:30 PM UTC

brown9-2 wrote: post #116

jmulho wrote: post #106

Earlier quoted context omitted.

I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the…

I was curious about a similar question to yours: if they had 66GB of RAM, why did going just slightly over the threshold cause such drastic paging for them - surely queries aren't touching all parts of their dataset equally? harryh's answer above about querying the user's entire history for each checkin answers this question though: http://news.ycombinator.com/item?id=1769909

IF your system is getting pounded with queries, and you suddenly have to page to serve those queries, the ability of your system to respond to queries and other operations goes WAY down, and the queries are just going to pile up, causing further swapping and load - it's a vicious cycle.

Re: MongoDB's lead developer: Foursquare outage post mortem

#180

Post by dedward » Sat, Oct 09, 2010, 5:33 PM UTC

sofuture wrote: post #158

robryan wrote: post #156

This may be a little naive seeing as I haven't used MongoDB or know anything about foursquare architecture, but wouldn't a better result have been just to drop requests that were hitting the disk? Sure some people would have been getting a sub optimal experience but arguably it's a lot better than not having your service up at all.

From my understanding, that 'routing' decision would have to take place inside of Mongo -- null routing DDoS packets on a network is one thing, dropping certain database requests on the fly inside a database server sounds a littttle bit tricker.

Tunable concurrency control for the number of incoming requests would have been good - before they even hit mongo.

Then you can just dial things down a bit (making customers wait a bit) and let the system recover, and tune things back to the point of optimal behaviour, rather than letting things overload. (You set your concurrency limits at a point where you know the system is just beginning to slow down but is still performing satisfactorily)