Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Root cause analysis: significantly elevated error rates on 2019‑07‑10

stripe.com

Post reply on HN
Print view View original

61–70 of 114 posts

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#61

Post by hyperpape »

scott00 wrote: post #47
sb8244 wrote: post #22

Earlier quoted context omitted.

I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code. However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine…

A rollback without understanding is definitely risky. An uninformed rollback is one of the factors that killed Knight Capital Group in 2012. For those not familiar, the actual problem was they failed to update one of a cluster of eight servers, and the server on the old version was making bad trades. They attempted to mitigate with a rollback, which made all eight servers start to make bad trades. In the end they los…

Knight Capital also didn't know what version of software their servers were running, didn't know which servers were originating the bad requests, had abandoned code still in the codebase, and reused flags that controlled that abandoned code (another summary: https://sweetness.hmmz.org/2013-10-22-how-to-lose-172222-a-s...). I'm not sure what you can infer about the risk of a rollback in a less crazy environment.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#63

Post by lethain »

aeyes wrote: post #48
lethain wrote: post #38

Earlier quoted context omitted.

(Stripe infra lead here) This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)

You might want to clarify this in the post. To me it reads like you knowingly had degraded infra for days leading up to an incident which might have been preventable had you recovered this instances.

Thanks for the suggestion, we’re adding a clarifying note to the report’s timeline.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#64

Post by tschwimmer »

tus88 wrote: post #56
dps wrote: post #34

Earlier quoted context omitted.

(Stripe CTO here) Thanks for the questions. We have testing procedures and deploy mechanisms that enable us to ship hundreds of deploys a week safely, including many which touch our infrastructure. For example, we do a fleetwide version rollout in stages with a blue/green deploy for typical changes. In this case, we identified a specific code path that we believed had a high potential to cause a follow-up incident so…

> ship hundreds of deploys a week safely That seems like a lot of change in a week, or does deploys mean something else like customer websites being deployed?

They very likely have continuous deployment. So each change could potentially be released as a separate deploy. If the changes have changed to the data model, they gotta run a migration. So hundreds seems reasonable to me.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#65

Post by Thaxll »

vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

They're also much simpler and the system behind payment solution didn't changed that much in the last 10 years.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#67

Post by jasonjei »

vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

That’s a great point. In spite of technical changes such as Apple Pay/Android Pay, chip cards, and so on, I can never recall an instance when I was unable to use a credit card globally. It seems most failures to running a credit card are pretty localized, too, and never at the interchange level...

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#68

Post by wallflower »

vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

Mainframes.

> Visa, for example, uses the mainframe to process billions of credit and debit card payments every year.

> According to some estimates, up to $3 trillion in daily commerce flows through mainframes.

https://www.share.org/blog/mainframe-matters-how-mainframes-...

https://blog.syncsort.com/2018/06/mainframe/9-mainframe-stat...

https://www.ibm.com/it-infrastructure/servers/mainframes

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#69

Post by londons_explore »

vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

Both have had plenty of downtime:

https://www.ft.com/content/1fd2a066-860f-11e8-a29d-73e3d4545...

I suspect they sometimes 'fail open' (ie. allow all payments through and reconcile later) too.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#70

Post by londons_explore »

Thaxll wrote: post #65
vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

They're also much simpler and the system behind payment solution didn't changed that much in the last 10 years.

They are also miles behind on features customers want...

For example:

* My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc.

* Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck?

* Why can't I use my mastercard to pay another mastercard user directly?

* Why hasn't mastercard produced a '2 factor' for card payments rather than forcing every bank to implement their own?

* Why can't I buy a dual Mastercard/Visa/Other card, which works with merchants who are picky and will only accept one or the other?

* Why are we still issuing bits of plastic in the digital age anyway?

* Why don't the cards have a microusb plug on one edge, or NFC to plug into a phone or computer to log in, to act as an identity card, to authenticate or make payments, or anything else other companies issue smartcards for?

* Why don't mastercard work with mobile providers to issue cards that you can spend your pay-as-you-go balance with, turning a mobile provider into a bank.

It seems mastercards business is 'stuck', and there are opportunities to innovate all around them, but they won't.

Post reply on HN