Root cause analysis: significantly elevated error rates on 2019‑07‑10 • Hacker News Board

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#101

Post by raverbashing » Sat, Jul 13, 2019, 8:28 AM UTC

jasonjei wrote: post #67

vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

That’s a great point. In spite of technical changes such as Apple Pay/Android Pay, chip cards, and so on, I can never recall an instance when I was unable to use a credit card globally. It seems most failures to running a credit card are pretty localized, too, and never at the interchange level...

I suspect there's a lot of caching involved as well. When making a purchase you probably don't need all the info to go all the way to the bank and back.

Stolen/lost cards can simply be flagged in a master db/table and can be rejected quickly for example.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#102

Post by raverbashing » Sat, Jul 13, 2019, 8:43 AM UTC

ashelmire wrote: post #40

laCour wrote: post #8

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

If you can think of every possible failure and create monitoring and reporting for it before it happens, then you're the best dev on the planet.

And then filtering for a lot of crap and false alarms the tools and supporting infrastructure throws

I kinda lost count of how many times Nagios barfed itself and reported an error while the application was fine

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#103

Post by zby » Sat, Jul 13, 2019, 10:02 AM UTC

So the article identifies a software bug and a software/config bug as the root cause. That sounds a bit shallow for such a high visibility case - I was expecting something like the https://en.wikipedia.org/wiki/5_Whys method with subplots on why the bugs where not caught in testing. By the way I only clicked on it because I was hoping it would be an occasion to use the methods from http://bayes.cs.ucla.edu/WHY/ - alas no - it was too shallow for that.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#104

Post by lathiat » Sat, Jul 13, 2019, 1:30 PM UTC

nulbyte wrote: post #93

segmondy wrote: post #81

Earlier quoted context omitted.

No they don't. If I sell a diamond ring for $20k and Visa passes that the card is valid but it's not. The buyer just got a free $20k ring. Card could be expired, cancelled, or not have enough balance. The merchant must be paid, their processor has to pay them, the bank that issued the card must provide that credit until the card holder pays it back. If the card was expired or a card with a $10 balance. The card holde…

Actually, merchants, acquirers, and issuers can do this. It happens sometimes. When it happens, other limits come into play downstream, such as terminal configuration. There are separate offline limits, and it is unlikely they would set it that high, so a $20,000 offline charge would be declined, even if a lesser charge would be approved, stored, and processed later. As for expired cards, the expiration date is on th…

It's even done regularly in some cases, various US airlines take "offline" card transactions and process them later for food & drinks.

There are of course limits on how big an offline transaction you can take intentionally or unintentionally and probably the airline wears the full cost of failed transactions in this case.

Doesn't matter that much when its for a $5 coffee, plus they know who you were if they really wanted to chase it down.

And as mentioned electronic terminals absolutely have automatic offline modes also.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#105

Post by Terretta » Sat, Jul 13, 2019, 1:37 PM UTC

wereHamster wrote: post #99

andrewg wrote: post #89

Earlier quoted context omitted.

Specifically they run IBM zTPF on their mainframes, which is also used by airlines. Some installations have uptimes measured in decades. https://www.ibm.com/it-infrastructure/z/transaction-processi...

It's rarely the hardware that fails, it's more often due to software. So I wonder what the software that's running on mainframes does differently than the software that's written for ordinary computers.

> So I wonder what the software that's running on mainframes does differently than the software that's written for ordinary computers.

Not change.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#106

Post by Terretta » Sat, Jul 13, 2019, 1:49 PM UTC

segmondy wrote: post #81

londons_explore wrote: post #69

Earlier quoted context omitted.

Both have had plenty of downtime: https://www.ft.com/content/1fd2a066-860f-11e8-a29d-73e3d4545... I suspect they sometimes 'fail open' (ie. allow all payments through and reconcile later) too.

No they don't. If I sell a diamond ring for $20k and Visa passes that the card is valid but it's not. The buyer just got a free $20k ring. Card could be expired, cancelled, or not have enough balance. The merchant must be paid, their processor has to pay them, the bank that issued the card must provide that credit until the card holder pays it back. If the card was expired or a card with a $10 balance. The card holde…

> No they don’t.

On the contrary, I developed early merchant and payment gateway tech, and they absolutely do. The scenario you describe is extraordinarily rare, which allows an arbitrage between CAP perfection and customer satisfaction.

On a separate note, at any given time, some parts of our national payments ecosystem are “down”. There are enough players involved you have an appearance of resilience.

You can see this in a mall, when one store’s card swipe terminals are down, and another’s are not, and almost never happens that all the stores are down at the same time.

You can think of all these other players as an incidental circuit breaker pattern upstream of Visa.

VisaNet itself is surprisingly unscaled, capable of only about 24,000 transactions per second. Twenty years ago, our gateway would hit 15,000 transactions per second real world use. To do that, we scattered/gathered across many independent paths into card networks and various merchant banks.

https://usa.visa.com/content/dam/VCOM/download/corporate/med...

https://www.capgemini.com/wp-content/uploads/2017/07/Domesti...

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#107

Post by jeremyjh » Sat, Jul 13, 2019, 1:53 PM UTC

segmondy wrote: post #81

londons_explore wrote: post #69

Earlier quoted context omitted.

Both have had plenty of downtime: https://www.ft.com/content/1fd2a066-860f-11e8-a29d-73e3d4545... I suspect they sometimes 'fail open' (ie. allow all payments through and reconcile later) too.

No they don't. If I sell a diamond ring for $20k and Visa passes that the card is valid but it's not. The buyer just got a free $20k ring. Card could be expired, cancelled, or not have enough balance. The merchant must be paid, their processor has to pay them, the bank that issued the card must provide that credit until the card holder pays it back. If the card was expired or a card with a $10 balance. The card holde…

They absolutely do, it is called "stand-in processing". I saw this while working in ATM at a major bank. The terminal operator (e.g. in our case, the ATM authorization system) can stand-in for the payment network when required. There are per-card number transaction limits that are well-defined in their contracts, and fraud liability can shift during this period of time. The payment network can also stand in for the issuer. In either case, once the network is restored all the authorization advices are forwarded.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#108

Post by zbentley » Sat, Jul 13, 2019, 2:18 PM UTC

zby wrote: post #103

So the article identifies a software bug and a software/config bug as the root cause. That sounds a bit shallow for such a high visibility case - I was expecting something like the https://en.wikipedia.org/wiki/5_Whys method with subplots on why the bugs where not caught in testing. By the way I only clicked on it because I was hoping it would be an occasion to use the methods from http://bayes.cs.ucla.edu/WHY/ - ala…

It is likely that this RCA was shallow because it was intended for everyone--including non-technical users, who (at least in my experience) tend to misinterpret or get confused by deep technical or systemic failure analysis.

It would be excellent if Stripe published a truly technical RCA, perhaps for distribution via their tech blog, so that folks like us could get a more complete understanding and what-not-to-do lesson (if the failing systems were based on non-proprietary technologies, that is).

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#109

Post by wbronitsky » Sat, Jul 13, 2019, 3:26 PM UTC

luizfelberti wrote: post #30

patio11 wrote: post #10

Earlier quoted context omitted.

I work at Stripe, on the marketing team, and assisted a bit here. My last major engineering work was writing the backend to a stock exchange. If anyone on HN knows anyone who has the sort of interesting life story where they both know what can cause a cluster election to fail and like writing about that sort of thing, we would eagerly like to make their acquaintance.

Maybe Kyle Kingsbury (aka @aphyr) is the person you are looking for? https://jepsen.io/services#consulting

Kyle used to work at Stripe and left. I don’t think he would come back unfortunately. That guy is absolutely amazing, especially with regards to distributes DBs and writing about them

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#110

Post by seanmcdirmid » Sat, Jul 13, 2019, 3:44 PM UTC

segmondy wrote: post #81

londons_explore wrote: post #69

Earlier quoted context omitted.

Both have had plenty of downtime: https://www.ft.com/content/1fd2a066-860f-11e8-a29d-73e3d4545... I suspect they sometimes 'fail open' (ie. allow all payments through and reconcile later) too.

No they don't. If I sell a diamond ring for $20k and Visa passes that the card is valid but it's not. The buyer just got a free $20k ring. Card could be expired, cancelled, or not have enough balance. The merchant must be paid, their processor has to pay them, the bank that issued the card must provide that credit until the card holder pays it back. If the card was expired or a card with a $10 balance. The card holde…

Credit cards are very asynchronous, going back to the days when carbon copy paper was used and no in time verification might have been involved at all.

Shop owners would even get a reward for snipping a bad credit user’s card in half (something that survives to this day only as a meme).