Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stripe – Outage postmortem

support.stripe.com

Post reply on HN
Print view View original

51–60 of 132 posts

Re: Stripe – Outage postmortem

#51

Post by eldavido »

jorgeortiz85 wrote: post #38

Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…

hi jorge

I'd actually applied to work at stripe about two years ago, you guys turned me down ;)

I was responsible for ops at a billion-device-scale mobile analytics company for about 1.5 years. Your tooling is far superior to anything we produced. I like the idea of a single source of truth describing the data model (code, tables, query patterns, etc.) a lot, and doubly-so that it's revision-controlled and available right alongside the code.

I think it's far from decided though, how much to involve human operators in processes like this. Judging from this answer, you seem to be on the extreme end of "automate everything". How then, I'm curious, do you train/communicate to developers what can be done safely vs. something that would cause i/o bottlenecks, slowdown, or other potentially production-impacting effects? Can you even predict these things accurately in advance? (Some of our worst outages were caused by emergent phenomena that only manifested at production scale, such as hitting packet throughput and network bandwidth limits on memcached -- totally unforseeable in a code-only test environment).

It sounds like you let developers request changes (a la "The Phoenix Project") but ops is responsible for final approval of the change? That actually sounds like a great system. Would love some elaboration on this.

In any case, great writeup and from one guy who's been there when the pager goes off to another, sounds like the recovery went pretty smoothly.

Re: Stripe – Outage postmortem

#52

Post by jewel »

protomyth wrote: post #45

So, and engineer can just submit a production change that an database operator will execute? What is the ceremony around this and does Stripe employ DBAs for production. What is the review process for a production change?

The change was to add a new index and then remove the old one. It's something that would have passed review.

The cause was a defect in the tooling. The requests weren't tied together in the way they were displayed and so the DBA removed the old index first.

Re: Stripe – Outage postmortem

#53

Post by radicalbyte »

Is it an option for you to maintain a set of tests which simulate behavior on the tables?

I've have great success doing this with datawarehouses (i.e. star schema, large tables with few writes). You run the tests after each index change on acceptance. It caught a errors.

For OLTP it's harder, you need to record some production workload and reply it. At your scale it's easier said than done, though.

Re: Stripe – Outage postmortem

#54

Post by kzhahou »

depsypher wrote: post #34
depsypher wrote: post #6

They clearly just need to add Devops. That'll fix everything.

Wow, message received HN. Don't make fun of Devops! It doesn't seem that long ago that it was hailed as the silver bullet that eliminates the "throw it over the wall" mentality that causes issues exactly like this one. This issue was caused by a failure in communication between team members. That communication is just as important as good engineering.

Your comment was obviously meant as commentary on the DevOps trend, but in fact devops is not so well-defined a trend to make a good target for sniping comments like this. I mean, people talk about "devops" today to mostly just say "we must increase investment in our infrastructure." No one's saying "devops will fix everything." So I don't think people downvoted you for cutting on devops per se, as much as for not presenting any clear POV.

It's not like Node, where evangelists DO go around saying it's the best thing ever and blindly ignore the problems with it. Now... there's a good target for snippy remarks! :-P

Re: Stripe – Outage postmortem

#55

Post by jorgeortiz85 »

eldavido wrote: post #51
jorgeortiz85 wrote: post #38

Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…

hi jorge I'd actually applied to work at stripe about two years ago, you guys turned me down ;) I was responsible for ops at a billion-device-scale mobile analytics company for about 1.5 years. Your tooling is far superior to anything we produced. I like the idea of a single source of truth describing the data model (code, tables, query patterns, etc.) a lot, and doubly-so that it's revision-controlled and available…

This is indeed a tricky balance. We want developers to iterate quickly, but we also want to understand the impact of production changes. With a small team and small sets of data, it's easy for everyone to understand the impact of changes and it's easy for modern hardware to hide inefficiencies. As we grow, the balance changes. It's harder for any one person to understand everything. It's also harder to hide inefficiencies with larger data sets.

We're always learning and improving. In order to scale, we'll need better ways to manage complexity and isolate failure. Our tools, patterns, and processes have changed quite a bit over the last few years, and they will continue to change. Ultimately, we want every Stripe employee to have the right information evident to them when they make decisions. This will be challenging, especially as we grow, but I'm excited to take on that challenge.

If you're still interested in working at Stripe, I'd encourage you to reapply! Our needs have changed quite a bit since you applied, and we're willing to reconsider candidates after a year has passed. Feel free to shoot me a resume: jorge@stripe.com

Re: Stripe – Outage postmortem

#56

Post by toomuchtodo »

eldavido wrote: post #51
jorgeortiz85 wrote: post #38

Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…

hi jorge I'd actually applied to work at stripe about two years ago, you guys turned me down ;) I was responsible for ops at a billion-device-scale mobile analytics company for about 1.5 years. Your tooling is far superior to anything we produced. I like the idea of a single source of truth describing the data model (code, tables, query patterns, etc.) a lot, and doubly-so that it's revision-controlled and available…

Shouldn't developers understand how a database change is going to impact an environment based on the code they've written?

Re: Stripe – Outage postmortem

#57

Post by perlgeek »

protomyth wrote: post #44
vruiz wrote: post #8

> At 00:08 UTC, our on-call engineer had been paged and had responded. At 00:10 UTC, we linked the API degradation to the removal of the index. I'm sure stripe has very good metrics of all their systems, nonetheless that's some rockstar level debugging skills.

We had a utility I wrote at one of the places I worked where you ran it against the database and it showed you all of the query plans running. It was easy with 15 seconds to see some non-indexed query and what was executing it. We used to run it after new version deployments to see if we had query problems. Tooling is very important.

As a stopgap solution, enabling the slow query log can also be very helpful.

Re: Stripe – Outage postmortem

#58

Post by xacaxulu »

gms wrote: post #29
jlarocco wrote: post #26

Earlier quoted context omitted.

I was thinking the same thing. If nothing else, I don't understand the role of "database operator" if they'll just blindly delete a critical index without thinking about it. Shouldn't that person have known better than anybody how critical the index was?

There is an open secret you may not be aware of: every startup is a shitshow on the inside.

Also top 20 financial institutions, USG orgs and places that store your healthcare and tax information ;-) I'd argue that a good number of startups these days (especially ones borne out of larger organizations with lots of combined experience) are way more capable of handling these issues with finesse and speed.

Re: Stripe – Outage postmortem

#60

Post by MichaelGG »

scurvy wrote: post #42

"Quick code fixes" almost always make the problem worse and just cause stress and anxiety. It's always better to simply tackle the root cause and fix that. Unless you're disabling a feature, don't push "quick code fixes". You'll pay for it later.

Except this seems to be counter factual? And the quick fixes were just temporary until the index was finished building. Seems like a perfectly fine thing to do to restore service during an outage, as long as they're rolled back or reviewed later.
Post reply on HN