Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

51–60 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#51

Post by IgorPartola »

kunley wrote: post #39
IgorPartola wrote: post #38

Earlier quoted context omitted.

I couldn't help but think of that exact issue. I suppose once compacting the data online is built in, this particular issue won't come up again. At the same time, when a machine is overloaded, often times you have even bigger problems. For example, if you are out of memory, you may not be able to create another SSH process to get at the box.

The situation antirez had in mind can be remedied by occasionally using a tool like vmtouch to steer what's in the OS cache.

Well, the other big advantage being that the OS has no idea about the particular kinds of data you want to store, whereas Redis has quite a bit more data to work with. But yes, vmtouch looks useful for these types of situations (thanks).

Re: MongoDB's lead developer: Foursquare outage post mortem

#52

Post by harryh »

redthrowaway wrote: post #19

The main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about t…

Thx for the kind words!

Re: MongoDB's lead developer: Foursquare outage post mortem

#53

Post by Nate75Sanders »

kunley wrote: post #39
IgorPartola wrote: post #38

Earlier quoted context omitted.

I couldn't help but think of that exact issue. I suppose once compacting the data online is built in, this particular issue won't come up again. At the same time, when a machine is overloaded, often times you have even bigger problems. For example, if you are out of memory, you may not be able to create another SSH process to get at the box.

The situation antirez had in mind can be remedied by occasionally using a tool like vmtouch to steer what's in the OS cache.

We need an "Ask HN" for cool tools. Maybe there's already a SO question about it. I hadn't heard of vmtouch and probably tons of other things people here use.

Re: MongoDB's lead developer: Foursquare outage post mortem

#54

Post by carbocation »

harryh wrote: post #42
fleitz wrote: post #40

Earlier quoted context omitted.

Why not keep the algorithm and process in batches? You don't need to have EVERY users checkin history in RAM at any moment. Throw the checkin on a queue, have a few processes that query a db for the checkin history and you're fine. Heck, if you delay the awards it's also a good excuse to throw the user an alert to come back to the site/app.

That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.

It also depends on your algorithms. Some algorithms are amenable to "running tallies," so a third possible approach would be to store and update various values based only on the aggregate past data plus the incremental data, instead of looking back through the entire history and recomputing when a new piece of data comes in. This of course depends on whether or not this is even theoretically possible with what you're computing.

Re: MongoDB's lead developer: Foursquare outage post mortem

#55

Post by chewbranca »

harryh wrote: post #37
jranck wrote: post #4

For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.

It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re-architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people…

I agree completely about the complexity of monitoring solutions for small teams with fluctuating applications. However, something as simple as htop running on an extra monitor would have alerted you of this issue long before downtime resulted.

Re: MongoDB's lead developer: Foursquare outage post mortem

#56

Post by alexpopescu »

While the details are very interesting, there are still many questions to be answered (on both sides):

- how difficult would be to bring up read-only replicas? (hopefully that should take much less than 11 hours + 6hours) - why the 3rd shard could accommodate only 5% of the data? - how can you plan capacity when using the "wrong" sharding? (basically leading to unpredictable distributions)

I have posted the rest of the questions here: http://nosql.mypopescu.com/post/1265191137/foursquare-mongod... as I hope to get some more answers.

Re: MongoDB's lead developer: Foursquare outage post mortem

#57

Post by kunley »

Nate75Sanders wrote: post #53
kunley wrote: post #39

Earlier quoted context omitted.

The situation antirez had in mind can be remedied by occasionally using a tool like vmtouch to steer what's in the OS cache.

We need an "Ask HN" for cool tools. Maybe there's already a SO question about it. I hadn't heard of vmtouch and probably tons of other things people here use.

The funny yet encouraging thing is that I'm just passing the wisdom: I've heard of vmtouch here on HN :)

Re: MongoDB's lead developer: Foursquare outage post mortem

#58

Post by [deleted] »

IgorPartola wrote: post #51
kunley wrote: post #39

Earlier quoted context omitted.

The situation antirez had in mind can be remedied by occasionally using a tool like vmtouch to steer what's in the OS cache.

Well, the other big advantage being that the OS has no idea about the particular kinds of data you want to store, whereas Redis has quite a bit more data to work with. But yes, vmtouch looks useful for these types of situations (thanks).

[deleted]

Re: MongoDB's lead developer: Foursquare outage post mortem

#59

Post by cullenking »

chewbranca wrote: post #55
harryh wrote: post #37

Earlier quoted context omitted.

It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re-architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people…

I agree completely about the complexity of monitoring solutions for small teams with fluctuating applications. However, something as simple as htop running on an extra monitor would have alerted you of this issue long before downtime resulted.

Awww come on, it's easy to say from an outside perspective. Regardless, the problem was handled well in the end, and we all get the benefit of understanding these limitations better. I think us tech people got the good side (information) out of this ordeal :)

Re: MongoDB's lead developer: Foursquare outage post mortem

#60

Post by chewbranca »

cullenking wrote: post #59
chewbranca wrote: post #55

Earlier quoted context omitted.

I agree completely about the complexity of monitoring solutions for small teams with fluctuating applications. However, something as simple as htop running on an extra monitor would have alerted you of this issue long before downtime resulted.

Awww come on, it's easy to say from an outside perspective. Regardless, the problem was handled well in the end, and we all get the benefit of understanding these limitations better. I think us tech people got the good side (information) out of this ordeal :)

I work on a team of 2 where I'm responsible for a handful of servers. I chimed in because I'm in a similar position, I've been looking for a monitoring solution for a while now. Things like nagios and zenoss are over kill, but lack of time has prevented me from finding an ideal solution. That said, I keep htop open and running at all times, and its saved my ass on more than one occasion. I say htop because of the color coding it provides, if things start going red it attracts my attention.
Post reply on HN