Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Root cause analysis: significantly elevated error rates on 2019‑07‑10

stripe.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#71

Post by londons_explore »

ssalazars wrote: post #45
bdamm wrote: post #29

Earlier quoted context omitted.

In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. When your customers are losing millions of dollars in the minutes you're down, mitigation would be the thing, and analysis can wait. All that is needed is enough forensic data so that testing in earnest to reproduce the condition in the lab can begin. Then get the customers back to…

I understand it. I've worked in AWS, and now in OCI, dealing with systems that affect hundreds-to-thousands of customers, which businesses are at stake. Mitigation is your top-priority. Bringing the system back to a good shape. If there needs to be follow-up actions, take the less-impactful steps to prevent another wave. If there was a deployment, roll-back. My concern here is, a deployment have been made months ago,…

> My concern here is, a deployment have been made months ago, and many other changes that could make things worse were introduced.

Make every bit of software in your stack export as a monitoring metric it's build date. Have an alert if any bit of software goes over 1 month old. Manually or automatically re-build and redeploy that software.

That prevents 'bit rot' meaning you daren't reduild or rollback something.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#72

Post by londons_explore »

Silhouette wrote: post #60
greenleafjacob wrote: post #46

Earlier quoted context omitted.

In that case you would probably still roll back to prevent further data corruption and restore the corrupted records from backups. There are certainly changes that cannot be rolled back such that the affected users are magically fixed, which is not what I am suggesting. In the context of mission critical systems, mitigation is usually strongly preferred. For example, the Google SRE book says the following: > Your fir…

In that case you would probably still roll back to prevent further data corruption and restore the corrupted records from backups. OK, but then what if it's new data being stored in real time, so there isn't any previous backup with the data in the intended form? In this case, we're talking about Stripe, which is presumably processing a high volume of financial transactions even in just a few minutes. Obviously there…

I think you guys are considering this from the wrong angle...

Rollbacks should always be safe. They should always be automatically tested. So a software release should do a gradual rollout (ie. 1, 10, 100, 1000 servers), but it should also restart a few servers with the old software version just to check a rollback still works.

The rollout should fail if health checks (including checking business metrics like conversion rates) on the new release or old release fails.

If only the new release fails, a rollback should be initiated automatically.

If only the old release fails, the system is in a fragile but still working state for a human to decide what to do.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#73

Post by londons_explore »

a13n wrote: post #49
conroy wrote: post #13

Earlier quoted context omitted.

MongoDB is the primary data store used at Stripe.

Really speaks volumes about how mature MongoDB has become considering how solid Stripe's reliability is.

MongoDB is a really scary database to use at scale.

It doesn't shard nicely. Failovers have rather nasty semantics that can cause nasty bugs in client side code. Performance cliffs abound.

If your datastore is anything over 1TB, I'd be using postgres, or if you can manage it something bigtable-like.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#74

Post by kortilla »

londons_explore wrote: post #70
Thaxll wrote: post #65

Earlier quoted context omitted.

They're also much simpler and the system behind payment solution didn't changed that much in the last 10 years.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

>Why are we still issuing bits of plastic in the digital age anyway?

Phones die.

If you don’t care, I suggest you look into Apple Pay or something similar. You’ll find many merchants that you won’t be able to pay.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#75

Post by kortilla »

laCour wrote: post #28
NikolaeVarius wrote: post #20

Earlier quoted context omitted.

In many HA setups, you're supposed to not have to care if any single thing goes down because it should auto recover The article said that the node stalled in a way that was unforseen which may have caused standard recovery mechanisms to silently fail.

Right, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.

The health check said it was ok. How would they know it needed to be recovered?

The fault was the bad health check. Not the process.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#76

Post by matthewmacleod »

kortilla wrote: post #74
londons_explore wrote: post #70

Earlier quoted context omitted.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

>Why are we still issuing bits of plastic in the digital age anyway? Phones die. If you don’t care, I suggest you look into Apple Pay or something similar. You’ll find many merchants that you won’t be able to pay.

Maybe in the US; here in the UK, contactless payment is now close to universal for any vendor who accepts cards. This suggests it’s eminently possible.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#77

Post by marbletiles »

londons_explore wrote: post #70
Thaxll wrote: post #65

Earlier quoted context omitted.

They're also much simpler and the system behind payment solution didn't changed that much in the last 10 years.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

Half of what you "want" is a quasi-dystopian nightmare.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#78

Post by zelon88 »

londons_explore wrote: post #70
Thaxll wrote: post #65

Earlier quoted context omitted.

They're also much simpler and the system behind payment solution didn't changed that much in the last 10 years.

They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard us…

Why would you want Target telling Mastercard that you bought Spongebob underwear and 1,968 calories worth of freeze pops?

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#79

Post by segmondy »

vjagrawal1984 wrote: post #62

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

They are not, they go down quite often. lol.
Post reply on HN