Stripe – Outage postmortem

Nicely done. Good job avoiding retrospectively blaming people and instead focusing on future system improvements.

(For those wondering why this is important, Sidney Dekker's "Field Guide to Understanding Human Error" is a mindblowing book.)

This is one of the best incident reports I think I've ever read, detailed, honest and without recrimination.

I'm reminded of something I read in a book a while back about aviation disasters "It's not the first problem that kills you, it's the second".

In this post the described system for change management is at least as good as any I've seen in production and yet a series of small problems got out of hand quickly.

I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the required execution order of two pieces of DDL. If process (like architecture) reflects organization, what does this mean about the organization that came up with this process? It's not a sloppy exactly. Is it overly-specialized? It reminds me of that despair.com poster "Meetings: none of us is as dumb as all of us" in that a process was invented that makes the group function below the competence level of any given member of the group.

Well the DB admin had just removed the index a few minutes before, and immediately the API times spiked. When you remove an index there is exactly one risk --slowdown-- and that's precisely what the admin noticed.