Coinbase Incident Post Mortem: June 25–26, 2019

oconnor663 wrote: post #4

kvlr wrote: post #2

TLDR: MongoDB

Sounds like a maintenance error and an application design error that led to a flood of traffic. Doesn't sound MongoDB specific.

I don't think there is enough information to come to a conclusion here, but I doubt your proposed failure mode.

Based on my experience with largish data in mongo, my guess would be that the database size was >> than ram and, due at least in part to mongo's design, when the master failed over the memory state of the new master didn't have the working set present in RAM. This lead to a huge inrush of disk IO resulting in all sorts of bad until the RAM state sorted itself out.

Re: Coinbase Incident Post Mortem: June 25–26, 2019

#7

Post by nocitrek » Sun, Jul 28, 2019, 9:14 PM UTC

Great post mortem. Well detailed, good work ethic there.

Re: Coinbase Incident Post Mortem: June 25–26, 2019

#8

Post by viraptor » Sun, Jul 28, 2019, 9:48 PM UTC

> We have ensured that failovers for this cluster may only be initiated during rare, scheduled downtime, when there will be no impact on customers.

I hope all their hardware crashes are also scheduled when there will be no impact... This seems a bit backwards - unless you constantly exercise the instantaneous failover, how do you know it works?

Edit: Actually it's worse - if you don't test the instant failover under a full load, how do you know it's still instant then?

Re: Coinbase Incident Post Mortem: June 25–26, 2019

#9

Post by H8crilA » Sun, Jul 28, 2019, 11:17 PM UTC

kvlr wrote: post #2

TLDR: MongoDB

Honestly I'm a bit surprised they don't use a more mature, fully managed solution. They can't have so much traffic to not justify getting some traditional relational database (like Oracle).

Re: Coinbase Incident Post Mortem: June 25–26, 2019

#10

Post by redis_mlc » Mon, Jul 29, 2019, 12:01 AM UTC

> We take uptime very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency

No you don't.

- If you did, you'd hire a DBA team and they would be familiar with the various jobs in your environment. But first your founders would have to have respect for Operations, which will take a dozen more major outages.

The other major Coinbase outages have also been database-related, namely missing indexes.

- If you did, you wouldn't be doing major database (or other production) changes at 3 pm in the afternoon.

So let's cut to the chase. You prioritize features over Operations, and as a result guinea-pig your users. Just like any other SF startup. So just admit that to your end-users.