Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's lead developer: Foursquare outage post mortem

groups.google.com

Post reply on HN
Print view View original

81–90 of 184 posts

Re: MongoDB's lead developer: Foursquare outage post mortem

#81

Post by daleharvey »

tripngroove wrote: post #76
jranck wrote: post #4

For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.

Disclaimer: I work at Cloudkick. We can help you all with these problems. Here's how fast/easy it is: 1. Create an account (~30 sec) 2. Add your cloud credentials (~45 sec) 3. Install the monitoring agent (~90 sec/node) 4. Create a CPU/Memory/Disk monitor, query-targeted all your servers (~60 sec) (example: "provider:EC2") 5. You get an email whenever the monitors you created reach the thresholds you set There are a…

While I usually frown on such blatant self promotion, I couldnt help but upvote this for being such a good advertisement, I want to use you now.

Re: MongoDB's lead developer: Foursquare outage post mortem

#82

Post by VladRussian »

harryh wrote: post #33
zaidf wrote: post #11

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a…

you can analyze the statistics of when/where specific people are most probable to check-in and pre-load (or even pre-calculate) the data in advance.

For example, starting at ~9pm the data about check-ins into library may be safely unloaded to disk (the probability of needing this data is low and reading from disk for whose rare cases would do just fine) and be replaced in memory with data about check-ins to clubs, etc...

Re: MongoDB's lead developer: Foursquare outage post mortem

#83

Post by nkallen »

harryh wrote: post #37
jranck wrote: post #4

For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.

It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re-architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people…

Suggestion:

In additions to hiring scalability experts (many will claim to be experts but most aren't), talk to your investors to find outside technical advisors who have worked on and are still working on similar problems.

Find advisors from Google, Facebook, and other places who have dealt with these issues. In my experience, you have as much to learn from people who have made mistakes as from people who have succeeded.

Also, in my judgement, you guys are a little too risk-tolerant for your significance (first major Scala 2.8 upgrade, largest MongoDB deploy...).

Re: MongoDB's lead developer: Foursquare outage post mortem

#84

Post by harryh »

nkallen wrote: post #80
harryh wrote: post #42

Earlier quoted context omitted.

That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.

Can you design the system such that you can turn off ("darkmode") features in an emergency? For example, in an emergency you can turn off badge-awarding as it is known to be RAM intensive (and degrade by seeking to disk). To the extent that you do this across the board, you'll have yet another tool to defend against being over capacity.

We have a framework for doing this sort of thing but it's only implemented around certain features so it doesn't provide us a ton of benefit at the moment. Obviously doing work to give us more control like this could be a big win, so we'll probably be doing some of that.

Re: MongoDB's lead developer: Foursquare outage post mortem

#86

Post by epochwolf »

doty wrote: post #44
seiji wrote: post #35

Earlier quoted context omitted.

They are dealing with web scale sharded NoSQL realtime geo scala. Old rules don't apply when 80% of the words describing your company didn't exist two years ago.

I honestly cannot tell whether or not that comment was serious.

Any comment with the words "web scale" should be considered a joke until proven otherwise. :)

Re: MongoDB's lead developer: Foursquare outage post mortem

#87

Post by mrkurt »

voxxit wrote: post #27

Why they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.

They actually said 4. Two shards with slaves for redundancy. It was apparently working fine until they lost track of what was going on. :)

Re: MongoDB's lead developer: Foursquare outage post mortem

#88

Post by jorgeortiz85 »

carbocation wrote: post #54
harryh wrote: post #42

Earlier quoted context omitted.

That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.

It also depends on your algorithms. Some algorithms are amenable to "running tallies," so a third possible approach would be to store and update various values based only on the aggregate past data plus the incremental data, instead of looking back through the entire history and recomputing when a new piece of data comes in. This of course depends on whether or not this is even theoretically possible with what you're…

This is probably feasible for some badges, but would be hard to do for all of them.

This approach also increases the complexity of adding new badges, which is undesirable for product and business reasons.

It could certainly help in some case though, and it's something we're considering.

Re: MongoDB's lead developer: Foursquare outage post mortem

#89

Post by wheels »

donaldc wrote: post #78
wheels wrote: post #72

Earlier quoted context omitted.

I don't think those are the same problem -- could you provide a link? The problem isn't paging, per se -- the paging system is doing exactly what it should be doing, paging in blocks off of the disk and into memory as they become hot, which for this use case is always. The problem is that you get fragmentation in your pages. If you allocate three records in a row that are 300 bytes, and then need to rewrite the first…

I don't think those are the same problem -- could you provide a link? You are correct, the actual problem is paging out LRU keys as opposed to memory holes. The issue is related but not the same. From http://antirez.com/post/what-is-wrong-with-2006-programming.... Multiply this for all the keys you have in memory and try visualizing it in your mind: These are a lot of small objects. What happens is simple to explain,…

What antirez is getting at there is actually a much harder (and more interesting) problem that could be generalized as something like "efficient data locality for mixed latency access".

However, assuming that all data must actually be in memory (as stated in the posted email), you don't actually solve the Mongo problem with more efficient organization of the data set, though compacting could be considered a sub-problem of the one that antirez describes.

But as I noted in my earlier comment, compacting wouldn't have actually solved their problems, it just would have delayed them. It's reasonable to ask if all of their data truly needs to be hot, but even there, you'd eventually hit diminishing returns as you approached the threshold where your active set couldn't fit in memory, and there smarter data organization wouldn't actually fix things once you started pulling chunks out for sharding; you'd still need to recompact.

Re: MongoDB's lead developer: Foursquare outage post mortem

#90

Post by rcoder »

t_crayford wrote: post #67
jasonwatkinspdx wrote: post #41

Building sharded systems isn't as simple as throwing consistent hashing into the code and calling it a day. You have to think carefully about what happens when nodes exceed capacity. There is good work in academia on distribution algorithms that gracefully handle reaching capacity (along with data center structure such as rack awareness) [1]. Alternately, if your algorithm doesn't handle a shard reaching capacity you…

I'm wondering if the decision to shard based on users was taken with data (ie, at that point did each user have a roughly similar number of check-ins); if not, hashing by that seems kind of fail for that kind of app.

From the email thread, it sounds like the decision to shard on UID was made mostly to increase locality of data, so that you didn't have to query more than one node to get a single user's data.

There's no silver bullet here. Hashing on insertion order would basically guarantee that writes would favor one node over another, which random hashes would force you to aggregate results from all available nodes for each query.

Post reply on HN