Stripe – Outage postmortem • Hacker News Board

Re: Stripe – Outage postmortem

#11

Post by nartz » Sat, Oct 10, 2015, 4:54 PM UTC

Good post mortem, just some thoughts, would love to hear what others think.

Shouldn't the app developers be in charge of deploying their code and making sure its working? It seems odd that they pass off a migration like this to a DBA to then go deploy randomly, or that they weren't around when it was happening to monitor it.

Also, it seems like there should have been a script that can be run that encapsulates the required dependencies, namely, don't drop an index before building the new one maybe? This should at least minimize the amount of context needed in a ticket.

Relying on fully correct context in tickets seems like it could be super error prone.

Re: Stripe – Outage postmortem

#12

Post by eldavido » Sat, Oct 10, 2015, 5:00 PM UTC

methehack wrote: post #7

I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…

I had this thought too. I'm actually surprised it was something this simple for an engineering org as competent and storied as Stripe, guess it proves Murphy gets all of us at one point or another.

The issue IMO is splitting an otherwise atomic procedure (creation/drop of index) into two change tickets. I'd be interested to know how the DDL was communicated to ops that led to its getting split.

Re: Stripe – Outage postmortem

#13

Post by vkat » Sat, Oct 10, 2015, 5:07 PM UTC

eldavido wrote: post #12

methehack wrote: post #7

I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…

I had this thought too. I'm actually surprised it was something this simple for an engineering org as competent and storied as Stripe, guess it proves Murphy gets all of us at one point or another. The issue IMO is splitting an otherwise atomic procedure (creation/drop of index) into two change tickets. I'd be interested to know how the DDL was communicated to ops that led to its getting split.

We do a ton of database updates regularly and we rely on a simple wiki put together that has all install instructions for a release which goes through sr engineers peer review. Seems like having a single tool/document for code and db updates could have mitigated this?

Re: Stripe – Outage postmortem

#14

Post by codegeek » Sat, Oct 10, 2015, 5:28 PM UTC

ohh. Could this mean that a payment may have failed ? I did have a failure of a payment around this time but it said "Card declined". So I hope it really was the card and not related to this incident.

Re: Stripe – Outage postmortem

#15

Post by [deleted] » Sat, Oct 10, 2015, 5:28 PM UTC

siliconc0w wrote: post #10

For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and fl…

[deleted]

Re: Stripe – Outage postmortem

#16

Post by rawnlq » Sat, Oct 10, 2015, 5:29 PM UTC

siliconc0w wrote: post #10

For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and fl…

Do people even test for perf issues in non-prod environments?

Re: Stripe – Outage postmortem

#17

Post by squiggy22 » Sat, Oct 10, 2015, 5:30 PM UTC

noir_lord wrote: post #5

This is one of the best incident reports I think I've ever read, detailed, honest and without recrimination. I'm reminded of something I read in a book a while back about aviation disasters "It's not the first problem that kills you, it's the second". In this post the described system for change management is at least as good as any I've seen in production and yet a series of small problems got out of hand quickly.

Totally agree. Came here to say pretty much the same thing. On a related note, highly recommend the Checklist manifesto (book) for devops.

Re: Stripe – Outage postmortem

#18

Post by toomuchtodo » Sat, Oct 10, 2015, 5:51 PM UTC

rawnlq wrote: post #16

siliconc0w wrote: post #10

For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and fl…

Do people even test for perf issues in non-prod environments?

I've found it to be extremely difficult to replicate production workloads in staging/development.

Re: Stripe – Outage postmortem

#19

Post by vruiz » Sat, Oct 10, 2015, 5:51 PM UTC

kzhahou wrote: post #9

vruiz wrote: post #8

> At 00:08 UTC, our on-call engineer had been paged and had responded. At 00:10 UTC, we linked the API degradation to the removal of the index. I'm sure stripe has very good metrics of all their systems, nonetheless that's some rockstar level debugging skills.

Well the DB admin had just removed the index a few minutes before, and immediately the API times spiked. When you remove an index there is exactly one risk --slowdown-- and that's precisely what the admin noticed.

That might be it, but by "on-call engineer" I understood it was someone else.

Re: Stripe – Outage postmortem

#20

Post by toomuchtodo » Sat, Oct 10, 2015, 5:52 PM UTC

methehack wrote: post #7

I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…

It's pretty easy to monday morning quarterback other org's choices or actions here. Every person posting here will have something in their org break at some point that was "obvious" to the rest of us.

EDIT: Someone, somewhere is going to have a bad day because they didn't know what you did. This is why sharing knowledge is so important. That's part of what HN exists for! Share what you know! Help improve open source tools! Help your fellow IT professional get a good night's sleep.