Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

91–100 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#91

Post by jmulho »

Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ?

Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM.

Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recently active 14 million users (4 gig / 300 bytes) will still enjoy instantaneous response.

Can someone help me out with the math here?

Re: MongoDB's lead developer: Foursquare outage post mortem

#92

Post by foobarbazetc »

harryh wrote: post #37
jranck wrote: post #4

For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.

It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re-architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people…

The fail here isn't just that you should have set up a shard 12 hours before or whatever. The other fail is that it seems (from the groups post) that you were relying on your data set only fitting in RAM but the underlying I/O system wasn't quite up to scratch in terms of being able to page in/out that additional 1GB of data.

It's not like your 66GB is hot all the time (or even big by any sort of measure), so that makes no sense to me, and that part of things needs to be further explained by 10gen and/or 4sq.

Re: MongoDB's lead developer: Foursquare outage post mortem

#94

Post by ww520 »

jmulho wrote: post #91

Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ? Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM. Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recent…

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory.

Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

Re: MongoDB's lead developer: Foursquare outage post mortem

#95

Post by chintan »

harryh wrote: post #33
zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

I suspect most of check-ins are happening in big cities such as NYC, SF etc. So some sort of "geo-based sharding" would be more helpful.

Also I recommend giving @naveen a separate shard. The guy just keeps roaming around in the city. Thank you.

Re: MongoDB's lead developer: Foursquare outage post mortem

#96

Post by JabavuAdams »

megablast wrote: post #75
SkyMarshal wrote: post #46

Earlier quoted context omitted.

Problem is lawsuits. Admitting guilt can and is used against you in court. Hence, US businesses don't apologize for anything. This is a refreshing exception.

I see it as a very American characteristic, from the people I have worked with before. Many Americans will never admit fault, never admit they do not know something, and always looking for someone else to blame. I think this comes from the culture of large companies. I have not seen this at smaller US companies, and other companies I have worked at. Of course, not all Americans, but I was surprised when I first saw t…

This is a large company / management culture thing, not a US thing. Also, it's a techie versus non-techie thing.

People in Canada and the US are more prone to admit mistakes than in more prestige / respect-oriented cultures. I.e. apparent lack of respect has its benefits.

Re: MongoDB's lead developer: Foursquare outage post mortem

#98

Post by rbranson »

jasonwatkinspdx wrote: post #34
zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

Facebook runs primarily out of ram via memcached. The last numbers I'm aware of were that they had about 200TB in memcache capacity [1]. They use a variety of data stores, but primarily sharded mysql. I don't have recent numbers there, but they were above 1000 master-master pairs as of 2008. While buying that much ram sound costly, that's only looking at capacity. Assuming typical 1u servers and common pricing at the…

Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of performance out of spindles, most "old school" DBAs will tell you that to get high throughput for a database you basically need to have RAM = database size.

Re: MongoDB's lead developer: Foursquare outage post mortem

#99

Post by silentbicycle »

ww520 wrote: post #94
jmulho wrote: post #91

Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ? Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM. Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recent…

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

Paging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.

Re: MongoDB's lead developer: Foursquare outage post mortem

#100

Post by darkhorse »

and out comes mongo's dirty little secret - you have to have enough ram in your boxes to hold not just all the data in ram, but all the indexes too, or it completely shits the bed.

putting hundreds of gigs of ram in a box isn't cheap.

are the foursquare folks considering rewriting with a traditional datastore like postgres and some memcached in front of it?

Post reply on HN