Stripe – Outage postmortem • Hacker News Board

Re: Stripe – Outage postmortem

#81

Post by BinaryIdiot » Sat, Oct 10, 2015, 9:03 PM UTC

toomuchtodo wrote: post #56

eldavido wrote: post #51

Earlier quoted context omitted.

hi jorge I'd actually applied to work at stripe about two years ago, you guys turned me down ;) I was responsible for ops at a billion-device-scale mobile analytics company for about 1.5 years. Your tooling is far superior to anything we produced. I like the idea of a single source of truth describing the data model (code, tables, query patterns, etc.) a lot, and doubly-so that it's revision-controlled and available…

Shouldn't developers understand how a database change is going to impact an environment based on the code they've written?

Yes they very much should! But in my, admittedly anecdotal, experience only the best / most senior ever do. Almost every junior or mid developer I've worked with (and a small handful of senior folks) not only have no idea how changes like this would impact the larger environment but many won't even care to look into it.

Re: Stripe – Outage postmortem

#82

Post by peoplelovelines » Sat, Oct 10, 2015, 9:09 PM UTC

Animats wrote: post #71

jacobolus wrote: post #63

Earlier quoted context omitted.

On the other hand, having a line out the door seems to sometimes be good marketing.

No. That went out decades ago. The Syufy chain of theaters (later Century) used to try to create lines by not opening up enough ticket windows. Then came video rental. Then came half-empty theaters. Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

Wrong. Clearly you've never lived in NYC, where people apparently use lines as a proxy for how good something is. Lots of other big cities too.

Also, box office revenue hasn't collapsed or anything, so I'm not sure what you mean with your example. In fact, come to think of it, I see people lining up for things multiple times every year at the theaters I frequent.

Re: Stripe – Outage postmortem

#83

Post by oalders » Sat, Oct 10, 2015, 9:34 PM UTC

peoplelovelines wrote: post #82

Animats wrote: post #71

Earlier quoted context omitted.

No. That went out decades ago. The Syufy chain of theaters (later Century) used to try to create lines by not opening up enough ticket windows. Then came video rental. Then came half-empty theaters. Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

Wrong. Clearly you've never lived in NYC, where people apparently use lines as a proxy for how good something is. Lots of other big cities too. Also, box office revenue hasn't collapsed or anything, so I'm not sure what you mean with your example. In fact, come to think of it, I see people lining up for things multiple times every year at the theaters I frequent.

When I see those long lines in front of [insert trendy food item] stores I think, "I'll have to try that one day when there's no lineup" but when that day actually comes, I've already forgotten about the place. Some of us do gauge popularity by lineups, but we also can't personally be bothered to stand in line.

Re: Stripe – Outage postmortem

#84

Post by onewaystreet » Sat, Oct 10, 2015, 9:39 PM UTC

potatosareok wrote: post #68

toomuchtodo wrote: post #20

Earlier quoted context omitted.

It's pretty easy to monday morning quarterback other org's choices or actions here. Every person posting here will have something in their org break at some point that was "obvious" to the rest of us. EDIT: Someone, somewhere is going to have a bad day because they didn't know what you did. This is why sharing knowledge is so important. That's part of what HN exists for! Share what you know! Help improve open source…

I get that but I'm wondering who is this postmortem written for? For other engineers? Not entirely - it's seems to be written partly as PR piece. In that case I don't need to basically congratulate Stripe on messing up and then posting PR piece on how they messed up, especially when it's such a trivial mistake (by that I mean it's not anything technically interesting for what went wrong). I guess I'll concede that wh…

> I get that but I'm wondering who is this postmortem written for?

For their customers.

> In that case I don't need to basically congratulate Stripe on messing up and then posting PR piece on how they messed up, especially when it's such a trivial mistake (by that I mean it's not anything technically interesting for what went wrong).

Few screw-ups are ever that technically interesting. The point of a postmortem isn't to be interesting, it's to explain what went wrong, why, and what you are doing to prevent it from happening again in the future.

Re: Stripe – Outage postmortem

#85

Post by ryporter » Sat, Oct 10, 2015, 9:46 PM UTC

People seem to be focusing on the fact that the two operations should have been linked, but I think they actually should have been further separated. The old index presumably could have been left around for days, at the cost of some performance, and only deleted once they were sure that the new index was successfully being used by all production code.

While splitting these two changes may be silly in the case of a simple index change, I think that it's a good general policy to only deploy the minimal set of changes to a production database at once. On the production system I manage (admittedly, much simpler, and many orders of magnitude smaller), I always deploy changes in two stages, with the second stage generally deployed about a week later, once I'm sure that only new code will be running against the database.

This comment is not meant to second guess the Stripe developers (who produced a great postmortem), but to suggest another possible remediation.

Re: Stripe – Outage postmortem

#86

Post by brown9-2 » Sat, Oct 10, 2015, 9:55 PM UTC

radicalbyte wrote: post #53

Is it an option for you to maintain a set of tests which simulate behavior on the tables? I've have great success doing this with datawarehouses (i.e. star schema, large tables with few writes). You run the tests after each index change on acceptance. It caught a errors. For OLTP it's harder, you need to record some production workload and reply it. At your scale it's easier said than done, though.

If the change that was introduced required creating a new index, it's unlikely that a test would have though to remove the now-outdated index or testing performance if the create-new/remove-old operations were reversed.

Re: Stripe – Outage postmortem

#87

Post by chrisabrams » Sat, Oct 10, 2015, 10:00 PM UTC

Animats wrote: post #71

jacobolus wrote: post #63

Earlier quoted context omitted.

On the other hand, having a line out the door seems to sometimes be good marketing.

No. That went out decades ago. The Syufy chain of theaters (later Century) used to try to create lines by not opening up enough ticket windows. Then came video rental. Then came half-empty theaters. Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

You've clearly never been to a big city.

Re: Stripe – Outage postmortem

#88

Post by chrisabrams » Sat, Oct 10, 2015, 10:00 PM UTC

peoplelovelines wrote: post #82

Animats wrote: post #71

Earlier quoted context omitted.

No. That went out decades ago. The Syufy chain of theaters (later Century) used to try to create lines by not opening up enough ticket windows. Then came video rental. Then came half-empty theaters. Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

Wrong. Clearly you've never lived in NYC, where people apparently use lines as a proxy for how good something is. Lots of other big cities too. Also, box office revenue hasn't collapsed or anything, so I'm not sure what you mean with your example. In fact, come to think of it, I see people lining up for things multiple times every year at the theaters I frequent.

Yeah, here in NYC people prefer to go to places where they wait in line. No line == no good.

Re: Stripe – Outage postmortem

#89

Post by lacksconfidence » Sat, Oct 10, 2015, 10:03 PM UTC

jorgeortiz85 wrote: post #38

Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…

have you considered integrating index statistics into these changes? To take an example from mysql, there is the INDEX_STATISTICS table in information_schema that contains the current number of rows read from the index. Checking this twice with a one minute interval before applying the index drop could have shown that the index was under heavy usage, and might require human intervention.

Re: Stripe – Outage postmortem

#90

Post by geofft » Sat, Oct 10, 2015, 10:07 PM UTC

depsypher wrote: post #34

depsypher wrote: post #6

They clearly just need to add Devops. That'll fix everything.

Wow, message received HN. Don't make fun of Devops! It doesn't seem that long ago that it was hailed as the silver bullet that eliminates the "throw it over the wall" mentality that causes issues exactly like this one. This issue was caused by a failure in communication between team members. That communication is just as important as good engineering.

I would interpret the message as "Don't make fun," not "Don't make fun of Devops." If you have a technical point to make, being sarcastic and curt is one of the worst possible ways to accurately convey that technical point.

Furthermore, if Stripe already uses devops, then your statement adds nothing to the conversation (maybe this follow-up comment would have, but your original comment didn't), and if they don't, then commentary about a practice that wasn't being used is the definition of an off-topic comment. So in either case, your comment is worthy of downvotes, regardless of the merits of its topic.

I don't think I've ever seen anyone downvoted on HN for being opposed to established wisdom with good reason. I've often seen people downvoted who were being opposed to either established or non-established wisdom in a way that doesn't contribute productively to the conversation. They weren't being downvoted for their beliefs, but for their lack of productive contribution to the conversation.