Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Root cause analysis: significantly elevated error rates on 2019‑07‑10

stripe.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#81

Post by segmondy »

londons_explore wrote: post #69
vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

Both have had plenty of downtime: https://www.ft.com/content/1fd2a066-860f-11e8-a29d-73e3d4545... I suspect they sometimes 'fail open' (ie. allow all payments through and reconcile later) too.

No they don't. If I sell a diamond ring for $20k and Visa passes that the card is valid but it's not. The buyer just got a free $20k ring. Card could be expired, cancelled, or not have enough balance. The merchant must be paid, their processor has to pay them, the bank that issued the card must provide that credit until the card holder pays it back. If the card was expired or a card with a $10 balance. The card holder will refuse, it get's really mess fast. Visa is not willing to assume such risk, they simply provide a network. If it goes down, it goes down and everyone on their network is screwed.

When dispute is at play, it's a hot potato that no one wants to hold between the merchant, processor, ISO, sales agent & bank. The card networks have been smart to eliminate themselves from that step.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#82

Post by Jorsiem »

throwaway3489 wrote: post #58
lethain wrote: post #38

Earlier quoted context omitted.

(Stripe infra lead here) This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)

I am a curious and very amateur person, but do you think that if "100%" uptime were your goal, this: "[Three months prior to the incident] We upgraded our databases to a new minor version that introduced a subtle, undetected fault in the database’s failover system." could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead re…

How do you have a ATM thats not networked?

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#83

Post by throwaway3491 »

Jorsiem wrote: post #82
throwaway3489 wrote: post #58

Earlier quoted context omitted.

I am a curious and very amateur person, but do you think that if "100%" uptime were your goal, this: "[Three months prior to the incident] We upgraded our databases to a new minor version that introduced a subtle, undetected fault in the database’s failover system." could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead re…

How do you have a ATM thats not networked?

Same user (sorry I guess I didn't enter my password carefully as I can't log in.)

Well I mean they're not exactly on the Internet with an IP address and no firewall, are they? (Or they would have been compromised already.)

Whatever it is, it must be separated off as an "insecure enclave".

So that's why I'm wondering about this technique. You don't just miss out on security updates, you miss performance and architecture improvements, too, if you stop upgrading.

But can that be the path toward 100% uptime? Known bad and out of date configurations, carefully maintained in a brittle known state?

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#84

Post by runevault »

gtirloni wrote: post #15
laCour wrote: post #8

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

In this environment: Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes. having a few nodes down is perfectly acceptable. I guess they would have had an alert if the number of down nodes exceeded some threshold.

this case that doesn't sound like it was the issue, it was the lack of promotion of new master due to the bug in the system in terms of shard promotion.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#85

Post by xyzzy_plugh »

zelon88 wrote: post #78
londons_explore wrote: post #70

Earlier quoted context omitted.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

Why would you want Target telling Mastercard that you bought Spongebob underwear and 1,968 calories worth of freeze pops?

Why do you believe this isn't already the case?

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#86

Post by arkades »

marbletiles wrote: post #77
londons_explore wrote: post #70

Earlier quoted context omitted.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

Half of what you "want" is a quasi-dystopian nightmare.

Don’t be such a pessimist. There’s nothing “quasi” about it.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#87

Post by Silhouette »

londons_explore wrote: post #72
Silhouette wrote: post #60

Earlier quoted context omitted.

In that case you would probably still roll back to prevent further data corruption and restore the corrupted records from backups. OK, but then what if it's new data being stored in real time, so there isn't any previous backup with the data in the intended form? In this case, we're talking about Stripe, which is presumably processing a high volume of financial transactions even in just a few minutes. Obviously there…

I think you guys are considering this from the wrong angle... Rollbacks should always be safe. They should always be automatically tested. So a software release should do a gradual rollout (ie. 1, 10, 100, 1000 servers), but it should also restart a few servers with the old software version just to check a rollback still works. The rollout should fail if health checks (including checking business metrics like convers…

This is one of those ideas that looks simple enough until you actually have to do it, and then you realise all the problems with it.

For example, in order to avoid any possibility of data loss at all using such a system, you need to continue running all of your transactions through the previous version of your system as well as the new version until you're happy that the performance of the new version is satisfactory. In the event of any divergence you probably need to keep the output of the previous version but also report the anomaly to whoever should investigate it.

But then if you're monitoring your production system, how do you make that decision about the performance of the new version being acceptable? If you're looking at metrics like conversion rates, you're going to need a certain amount of time to get a statistically significant result if anything has broken. Depending on your system and what constitutes a conversion, that might take seconds or it might take days. And you can only make a single change, which can therefore be rolled back to exactly the previous version without any confounding factors, during that whole time.

And even if you provide a doubled-up set of resources to run new versions in parallel and you insist on only rolling out a single change to your entire system during a period of time that might last for days in case extended use demonstrates a problem that should trigger an automatic rollback, you're still only protecting yourself against problems that would show up in whatever metric(s) you chose to monitor. The real horror stories are very often the result of failure modes that no-one anticipated or tried to guard against.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#88

Post by D-Coder »

marbletiles wrote: post #77
londons_explore wrote: post #70

Earlier quoted context omitted.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

Half of what you "want" is a quasi-dystopian nightmare.

If this is what he wants for himself, it's not dystopian, it's personal info.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#89

Post by andrewg »

wallflower wrote: post #68
vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

Mainframes. > Visa, for example, uses the mainframe to process billions of credit and debit card payments every year. > According to some estimates, up to $3 trillion in daily commerce flows through mainframes. https://www.share.org/blog/mainframe-matters-how-mainframes-... https://blog.syncsort.com/2018/06/mainframe/9-mainframe-stat... https://www.ibm.com/it-infrastructure/servers/mainframes

Specifically they run IBM zTPF on their mainframes, which is also used by airlines. Some installations have uptimes measured in decades.

https://www.ibm.com/it-infrastructure/z/transaction-processi...

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#90

Post by tluyben2 »

kortilla wrote: post #74
londons_explore wrote: post #70

Earlier quoted context omitted.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

>Why are we still issuing bits of plastic in the digital age anyway? Phones die. If you don’t care, I suggest you look into Apple Pay or something similar. You’ll find many merchants that you won’t be able to pay.

In the west, but in China you pay everything with wechat or alipay and similar solutions are popping up in Asia in every country with success. But in China it is accepted everywhere. And when it is not it still is because usually the shop cashiers will use their own phone to complete the transaction.
Post reply on HN