Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stripe – Outage postmortem

support.stripe.com

Post reply on HN
Print view View original

71–80 of 132 posts

Re: Stripe – Outage postmortem

#71

Post by Animats »

jacobolus wrote: post #63
Animats wrote: post #49

Earlier quoted context omitted.

There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expe…

On the other hand, having a line out the door seems to sometimes be good marketing.

No. That went out decades ago. The Syufy chain of theaters (later Century) used to try to create lines by not opening up enough ticket windows. Then came video rental. Then came half-empty theaters.

Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

Re: Stripe – Outage postmortem

#72

Post by tempestn »

devit wrote: post #67
jorgeortiz85 wrote: post #38

Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…

Why not just use simple version controlled database migrations, and testing them in a test environment?

Generally you want your database migrations described in a straightforward manner for development; the migrations will contain a straightforward change from old to new (and back). With a live (busy) production database, it is often necessary to handle things differently to maintain up-time.

As a simple example, to make an atomic change to a write-only table, you could create a copy of the table, alter the copy as necessary, then in a single rename operation, rename the live table to '_old' and the '_new' table to live. You most likely would not want to add two additional table schema and all of those steps to your development database operations.

It's entirely possible that they could capture what is done in production as migrations, and test them first, but it would still likely be separate from what the application developers are working with.

Re: Stripe – Outage postmortem

#73

Post by rjbrock »

vostro_mf wrote: post #59

Sigh, yet another example of hot shot teams using MongoDB just because it's new and sexy. Existing, established tools such as Oracle and Postgres would have offered lots of ways of avoiding such a problem.

I am a huge fan of postgres, but removing an index on postgres like this (especially if it is the primary means of querying a large table) would have the same effects.

This is just a failure of the tool that executes indexing operations, and not of the db itself

Re: Stripe – Outage postmortem

#74

Post by protomyth »

shazow wrote: post #50
protomyth wrote: post #44

Earlier quoted context omitted.

We had a utility I wrote at one of the places I worked where you ran it against the database and it showed you all of the query plans running. It was easy with 15 seconds to see some non-indexed query and what was executing it. We used to run it after new version deployments to see if we had query problems. Tooling is very important.

That sounds really handy, any chance it's publicly available or you can make it so?

No, I couldn't release it due to not having the source or being in a position to ask permission. It was for Sybase but the same thing can be achieved in MS SQL by reading some system tables. It really wasn't that hard of a thing to write.

Re: Stripe – Outage postmortem

#75

Post by protomyth »

jewel wrote: post #52
protomyth wrote: post #45

So, and engineer can just submit a production change that an database operator will execute? What is the ceremony around this and does Stripe employ DBAs for production. What is the review process for a production change?

The change was to add a new index and then remove the old one. It's something that would have passed review. The cause was a defect in the tooling. The requests weren't tied together in the way they were displayed and so the DBA removed the old index first.

That's not a tooling defect, that's a process and personal knowledge of the system defect. No DBA should remove an index without knowing what is replacing it or that code got deployed to make it obsolete. Checking if that index is currently being used would be a minimum. If a DBA cannot stop a deploy you have process problems.

Re: Stripe – Outage postmortem

#76

Post by devit »

tempestn wrote: post #72
devit wrote: post #67

Earlier quoted context omitted.

Why not just use simple version controlled database migrations, and testing them in a test environment?

Generally you want your database migrations described in a straightforward manner for development; the migrations will contain a straightforward change from old to new (and back). With a live (busy) production database, it is often necessary to handle things differently to maintain up-time. As a simple example, to make an atomic change to a write-only table, you could create a copy of the table, alter the copy as nec…

Development databases normally have a small amount of data, so migrations should execute instantly or nearly so no matter how complex they are.

Re: Stripe – Outage postmortem

#77

Post by toomuchtodo »

potatosareok wrote: post #68
toomuchtodo wrote: post #20

Earlier quoted context omitted.

It's pretty easy to monday morning quarterback other org's choices or actions here. Every person posting here will have something in their org break at some point that was "obvious" to the rest of us. EDIT: Someone, somewhere is going to have a bad day because they didn't know what you did. This is why sharing knowledge is so important. That's part of what HN exists for! Share what you know! Help improve open source…

I get that but I'm wondering who is this postmortem written for? For other engineers? Not entirely - it's seems to be written partly as PR piece. In that case I don't need to basically congratulate Stripe on messing up and then posting PR piece on how they messed up, especially when it's such a trivial mistake (by that I mean it's not anything technically interesting for what went wrong). I guess I'll concede that wh…

I would encourage you to accept that their post mortem was released in good faith, and its purpose is to both technical knowledge sharing as well as PR. I know I personally value technical organizations that are honest and forthcoming when things go south.

Re: Stripe – Outage postmortem

#78

Post by cookiecaper »

vostro_mf wrote: post #59

Sigh, yet another example of hot shot teams using MongoDB just because it's new and sexy. Existing, established tools such as Oracle and Postgres would have offered lots of ways of avoiding such a problem.

Although it's probably not directly relevant to this problem, I agree with you. MongoDB is the new MySQL; early on the scene and sexy, but at a real cost. There are other solutions doing the same things much better, and your life will be easier if you do your research before jumping into the sexy solution.

Companies should think carefully before introducing MongoDB (or any immature project) into critical production stacks.

Re: Stripe – Outage postmortem

#79

Post by cookiecaper »

raverbashing wrote: post #47
Animats wrote: post #33

What's so striking about this is that entire retail chains can be shut down by a problem in some cloud server. Starbucks had a server outage in April which caused stores to close.[1] There's a trend towards "hosted POS", where point of sale systems have to talk to the "cloud" to do anything, even handle cash. Until recently, most POS systems were running off a server in the manager's office, which communicated to ser…

Exactly this The "2.0 generation" (or was that 5 years ago?) is forgoing a lot of redundancy and safeguards that existed in the previous systems When did we assume a fast data link (and all the infrastructure behind it) would be available 24/7? At least do cash transactions and note them in a notebook

Yep, I'm not sure what the appeal here is. It just seems like companies are putting out less robust solutions because it saves them time and money and people are going for it because the company has sleek Silicon Valley marketing, regardless of the practicalities.

Re: Stripe – Outage postmortem

#80

Post by dang »

vostro_mf wrote: post #59

Sigh, yet another example of hot shot teams using MongoDB just because it's new and sexy. Existing, established tools such as Oracle and Postgres would have offered lots of ways of avoiding such a problem.

We detached this subthread from https://news.ycombinator.com/item?id=10366459 and marked it off-topic.
Post reply on HN