Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

121–130 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#121

Post by dirkgadsden »

look_lookatme wrote: post #114

While I understand this is foremost a monitoring and architecture (mongo and 4sq side...) issue, but I'm curious how dangerous it is to run such IO dependent systems on EC2. Would the prolonged downtime (due to shard migration, etc) been as severe if they were running on hardware? What if these two servers were on SSD RAID 0?

The duration of the downtime seems to have been dependent on not just disk throughput, but also network and CPU throughput. Eliot did mention that a large portion of the downtime was partly caused by the slowness of EBS. Making a rough estimate, I think the downtime would probably be about 2/3 of how long it was if it had been on an SSD RAID. However, then you run into the issue of having to maintain your own servers; which, judging by the fact that they are using EC2 and EBS so heavily, Foursquare does not have any desire whatsoever to manage their own infrastructure.

Re: MongoDB's lead developer: Foursquare outage post mortem

#122

Post by dirkgadsden »

harryh wrote: post #61
alexpopescu wrote: post #56

While the details are very interesting, there are still many questions to be answered (on both sides): - how difficult would be to bring up read-only replicas? (hopefully that should take much less than 11 hours + 6hours) - why the 3rd shard could accommodate only 5% of the data? - how can you plan capacity when using the "wrong" sharding? (basically leading to unpredictable distributions) I have posted the rest of t…

> how difficult would be to bring up read-only replicas? > (hopefully that should take much less than 11 hours > 6hours) Bringing up read only replicas would have been easy, but our appservers are not currently designed to read data from multiple replicas so it wouldn't have helped. We hope to make architectural changes to allow for this sort of thing in the future but aren't there yet. > - why the 3rd shard could ac…

What exactly is the rate like for transferring data across MongoDB instances on EBS? Was the overhead of going across EBS volumes the major factor in the length of the downtime?

The 60%/40% sharding distribution doesn't seem too bad in the big picture; are there any plans to be evening that out?

Re: MongoDB's lead developer: Foursquare outage post mortem

#123

Post by ww520 »

silentbicycle wrote: post #99
ww520 wrote: post #94

Earlier quoted context omitted.

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

Paging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.

The problem isn't the page granularity, which is just a convenient size for the general case and works pretty well in most cases. The problem is to not keep excessive data in memory; in this case, they are keeping ALL data in memory due to some architectural decisions.

OS caching and paging have work pretty well in keeping the frequently used data in memory and leave the rest on disk, and thus utilize the system resource more efficiently.

Re: MongoDB's lead developer: Foursquare outage post mortem

#124

Post by hartror »

there wrote: post #112

so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monito…

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

Re: MongoDB's lead developer: Foursquare outage post mortem

#125

Post by look_lookatme »

dirkgadsden wrote: post #121
look_lookatme wrote: post #114

While I understand this is foremost a monitoring and architecture (mongo and 4sq side...) issue, but I'm curious how dangerous it is to run such IO dependent systems on EC2. Would the prolonged downtime (due to shard migration, etc) been as severe if they were running on hardware? What if these two servers were on SSD RAID 0?

The duration of the downtime seems to have been dependent on not just disk throughput, but also network and CPU throughput. Eliot did mention that a large portion of the downtime was partly caused by the slowness of EBS. Making a rough estimate, I think the downtime would probably be about 2/3 of how long it was if it had been on an SSD RAID. However, then you run into the issue of having to maintain your own servers…

It's an interesting angle if that's the reason they are using EC2. Dedicated hosting might be more expensive, but hardware managed is pretty efficient at this point.

Also, given the vast number of EC2 instances they could afford, it seems counter intuitive to be running only two mongrel shards. If you are that stingy about spreading data around, you might as well be using dedicated.

Re: MongoDB's lead developer: Foursquare outage post mortem

#126

Post by hartror »

sjs wrote: post #70
chewbranca wrote: post #60

Earlier quoted context omitted.

I work on a team of 2 where I'm responsible for a handful of servers. I chimed in because I'm in a similar position, I've been looking for a monitoring solution for a while now. Things like nagios and zenoss are over kill, but lack of time has prevented me from finding an ideal solution. That said, I keep htop open and running at all times, and its saved my ass on more than one occasion. I say htop because of the col…

Nagios is pretty nice. It's dead easy to write custom monitors and clients are everywhere, there's even a Firefox extension. It requires a bit of learning to get going with but it's not so bad and the pay-off is big. That said I'm looking at monit too. I hear it's quite nice and has less of a learning curve.

I'll put a vote in for Zabbix as a good option, it has saved us more times than I can count.

Re: MongoDB's lead developer: Foursquare outage post mortem

#128

Post by hartror »

darkhorse wrote: post #100

and out comes mongo's dirty little secret - you have to have enough ram in your boxes to hold not just all the data in ram, but all the indexes too, or it completely shits the bed. putting hundreds of gigs of ram in a box isn't cheap. are the foursquare folks considering rewriting with a traditional datastore like postgres and some memcached in front of it?

Err you don't . . as the article states their usage and load pattern does. If you want the speed of keeping the DB in RAM you need the RAM available whether it is mongo, MySQL/NDB or memcache. (-1 for not RTFA)

Re: MongoDB's lead developer: Foursquare outage post mortem

#129

Post by swaits »

swaits wrote: post #62

Right, kick ass. Well, don't want to sound like a dick or nothin', but, ah... it says on your chart that you're fucked up. Ah, you talk like a fag, and your shit's all retarded. What I'd do, is just like... like... you know, like, you know what I mean, like...

Geez. No sense of humor. It's a line from Idiocracy. Hilarious.
Post reply on HN