Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stripe – Outage postmortem

support.stripe.com

Post reply on HN
Print view View original

91–100 of 132 posts

Re: Stripe – Outage postmortem

#91

Post by spudlyo »

lacksconfidence wrote: post #89
jorgeortiz85 wrote: post #38

Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…

have you considered integrating index statistics into these changes? To take an example from mysql, there is the INDEX_STATISTICS table in information_schema that contains the current number of rows read from the index. Checking this twice with a one minute interval before applying the index drop could have shown that the index was under heavy usage, and might require human intervention.

That was my thought as well, but this change was done by an Operator and not a DBA, who tend to be a bit more curious about these kinds of changes.

Re: Stripe – Outage postmortem

#92

Post by thrownaway2424 »

jacobolus wrote: post #63
Animats wrote: post #49

Earlier quoted context omitted.

There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expe…

On the other hand, having a line out the door seems to sometimes be good marketing.

You can have a line and good throughput at the same time. It's easy to develop a line. What you don't want is to have two espresso machines with three group heads each and two baristas just standing around because the counter is too slow.

Re: Stripe – Outage postmortem

#93

Post by dexterdog »

toomuchtodo wrote: post #56
eldavido wrote: post #51

Earlier quoted context omitted.

hi jorge I'd actually applied to work at stripe about two years ago, you guys turned me down ;) I was responsible for ops at a billion-device-scale mobile analytics company for about 1.5 years. Your tooling is far superior to anything we produced. I like the idea of a single source of truth describing the data model (code, tables, query patterns, etc.) a lot, and doubly-so that it's revision-controlled and available…

Shouldn't developers understand how a database change is going to impact an environment based on the code they've written?

You would think but far too many developers don't really know how databases work under load.

Re: Stripe – Outage postmortem

#94

Post by la6470 »

Why would someone delete a existing index without recreating it first? This is common sense for a DBA but not for our full stack engineers who have to know everything about everything g.

Re: Stripe – Outage postmortem

#95

Post by mst »

perlgeek wrote: post #35
methehack wrote: post #7

I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…

> It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the required execution order of two pieces of DDL. To me it seemed that it was a bug in the tooling that split the index change into two separate change requests. I'm sure a change request supports more than one piece of DDL, and it must have worked in the normal case, otherwise they w…

> Now that I think about it, this could happen to us as well. We have peer review for each database change, but only at the source level (so, definition of the schema); the actual commands for schema changes are usually generated automatically.

If you're using any of the DBIx::Class deployment tools they should be perfectly happy writing the DDL to disk and then running it from there, specifically to make it possible to audit the DDL as part of the commit that changed the result classes.

Generated != unauditable, especially when the tools are trying their best to co-operate :D

Re: Stripe – Outage postmortem

#97

Post by mst »

ryporter wrote: post #85

People seem to be focusing on the fact that the two operations should have been linked, but I think they actually should have been further separated. The old index presumably could have been left around for days, at the cost of some performance, and only deleted once they were sure that the new index was successfully being used by all production code. While splitting these two changes may be silly in the case of a si…

I had a similar thought but if it's that critical an index it's entirely possible they couldn't afford the write overhead to keep both copies around.

Re: Stripe – Outage postmortem

#98

Post by mst »

protomyth wrote: post #75
jewel wrote: post #52

Earlier quoted context omitted.

The change was to add a new index and then remove the old one. It's something that would have passed review. The cause was a defect in the tooling. The requests weren't tied together in the way they were displayed and so the DBA removed the old index first.

That's not a tooling defect, that's a process and personal knowledge of the system defect. No DBA should remove an index without knowing what is replacing it or that code got deployed to make it obsolete. Checking if that index is currently being used would be a minimum. If a DBA cannot stop a deploy you have process problems.

The post mortem specifically noted that it looked unused, presumably because the code in master now claimed to depend on the new index.

Re: Stripe – Outage postmortem

#99

Post by mst »

scurvy wrote: post #42

"Quick code fixes" almost always make the problem worse and just cause stress and anxiety. It's always better to simply tackle the root cause and fix that. Unless you're disabling a feature, don't push "quick code fixes". You'll pay for it later.

Given it was temporary and they said it led to the API running "slightly degraded", it sounds like disabling a feature is pretty much exactly what they did.

Re: Stripe – Outage postmortem

#100

Post by intopieces »

Animats wrote: post #71
jacobolus wrote: post #63

Earlier quoted context omitted.

On the other hand, having a line out the door seems to sometimes be good marketing.

No. That went out decades ago. The Syufy chain of theaters (later Century) used to try to create lines by not opening up enough ticket windows. Then came video rental. Then came half-empty theaters. Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

>Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

And yet Apple had a line 12+ hours before opening to release their 9th generation iPhone -- despite also taking orders online.

Hopdoddy in Austin has a line out the door for lunch and dinner; they even have a person dedicated to managing it, along with waitstaff to take drink orders

I think the line is alive and well in America for certain establishments.

Post reply on HN