Woot, a postmortem from Stripe! I've been asking for these for over a year and hopefully we'll see more from them in the future.
Stripe – Outage postmortem
21–30 of 132 posts
Re: Stripe – Outage postmortem
#22ohh. Could this mean that a payment may have failed ? I did have a failure of a payment around this time but it said "Card declined". So I hope it really was the card and not related to this incident.
Failures caused by this should have returned an api_error, a card_error (like card declined) would have been returned upstream from Stripe.
Re: Stripe – Outage postmortem
#23For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and fl…
Do people even test for perf issues in non-prod environments?
If it's a requirement to be within X performance then yes.
Re: Stripe – Outage postmortem
#24For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and fl…
Do people even test for perf issues in non-prod environments?
Yes you can do scaled down testing in non-prod. An example test may try 100,000 or 1,000,000 operations when in production you might see 100,000,000. So you won't catch subtle performance problems but you'll catch order of magnitude problems (i.e something that is O(n) vs O(log(n))
Although in this case the test for this change may just EXPLAIN a known query and ensure the output matches what is expected.
Re: Stripe – Outage postmortem
#25What is the point of having a human "database operator" who carries out simple tasks like deleting an index when it shows up in a work request log? Is this how most companies structure their dev teams?
If you have a human there to perform tasks, then it would seem natural to allow them "human advantages" such as the ability to communicate with the person who created the request, or the ability to have some of their own checks and balances before performing the index deletion (ex. let's take a look and see if this seems safe based on the current schema and codebase).
I am also surprised how easy it was for a single dev to make a request that subsequently results in the modification of a production database.
Re: Stripe – Outage postmortem
#26I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…
I was thinking the same thing.
If nothing else, I don't understand the role of "database operator" if they'll just blindly delete a critical index without thinking about it. Shouldn't that person have known better than anybody how critical the index was?
Re: Stripe – Outage postmortem
#27Earlier quoted context omitted.
Well the DB admin had just removed the index a few minutes before, and immediately the API times spiked. When you remove an index there is exactly one risk --slowdown-- and that's precisely what the admin noticed.
That might be it, but by "on-call engineer" I understood it was someone else.
Sure. I picture it as:
"Hey {on-call-engineer}, we're seeing a huge slowdown all of the sudden."
"Was there any change in the last few minutes?"
"We just deleted a database index."
"Yup, that'll do it. Gotta restore it."
Re: Stripe – Outage postmortem
#28Earlier quoted context omitted.
I had this thought too. I'm actually surprised it was something this simple for an engineering org as competent and storied as Stripe, guess it proves Murphy gets all of us at one point or another. The issue IMO is splitting an otherwise atomic procedure (creation/drop of index) into two change tickets. I'd be interested to know how the DDL was communicated to ops that led to its getting split.
We do a ton of database updates regularly and we rely on a simple wiki put together that has all install instructions for a release which goes through sr engineers peer review. Seems like having a single tool/document for code and db updates could have mitigated this?
How about automated migrations? At the scale of Stripe, you may want to have dedicated personnel keeping an eye on it, but had it been in the same SQL file, none of that would have happened...
Re: Stripe – Outage postmortem
#29I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…
I was thinking the same thing. If nothing else, I don't understand the role of "database operator" if they'll just blindly delete a critical index without thinking about it. Shouldn't that person have known better than anybody how critical the index was?
There is an open secret you may not be aware of: every startup is a shitshow on the inside.
Re: Stripe – Outage postmortem
#30> At 00:08 UTC, our on-call engineer had been paged and had responded. At 00:10 UTC, we linked the API degradation to the removal of the index. I'm sure stripe has very good metrics of all their systems, nonetheless that's some rockstar level debugging skills.
I'd hope one of the first things for DB slow down is the running queries (eg. `show processlist` in flavors of MySQL) and seeing the number of rows examined. Maybe your DB doesn't have that, so you see hopefully see long running queries and they are mostly the same one, or you can see the same queries in your DB's engine status (eg. `show innodb status` in MySQL). If you don't have rows examined from earlier, you can run whatever equivalent you have of `explain` on these long running queries that are happening a lot, and now you should find the number of rows it is examining (and probably that it is running a scan query and not indexed).
I mean, you only get these scars from actually working with your DB and hopefully in development / capacity planning stages, so maybe that makes you a rockstar. Not to take anything away from the Stripe engineer(s).