Root cause analysis: significantly elevated error rates on 2019‑07‑10 • Hacker News Board

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#111

Post by bdamm » Sat, Jul 13, 2019, 9:26 PM UTC

londons_explore wrote: post #71

ssalazars wrote: post #45

Earlier quoted context omitted.

I understand it. I've worked in AWS, and now in OCI, dealing with systems that affect hundreds-to-thousands of customers, which businesses are at stake. Mitigation is your top-priority. Bringing the system back to a good shape. If there needs to be follow-up actions, take the less-impactful steps to prevent another wave. If there was a deployment, roll-back. My concern here is, a deployment have been made months ago,…

> My concern here is, a deployment have been made months ago, and many other changes that could make things worse were introduced. Make every bit of software in your stack export as a monitoring metric it's build date. Have an alert if any bit of software goes over 1 month old. Manually or automatically re-build and redeploy that software. That prevents 'bit rot' meaning you daren't reduild or rollback something.

In a lot of environments this is a terrible idea. In private environments exposing build manifest information is a good idea, but not so that you can alert at 1 month. Where I work, software that's 2-3 years old is considered good - mature, tested, thoroughly operationalized, and understood by all who need to interact with it on a daily basis. Often, consistency of the user experience is better than being bug free.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#112

Post by redis_mlc » Sat, Jul 13, 2019, 11:58 PM UTC

throwaway3489 wrote: post #58

lethain wrote: post #38

Earlier quoted context omitted.

(Stripe infra lead here) This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)

I am a curious and very amateur person, but do you think that if "100%" uptime were your goal, this: "[Three months prior to the incident] We upgraded our databases to a new minor version that introduced a subtle, undetected fault in the database’s failover system." could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead re…

> could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead relying on containing it as a "known" vulnerable database?

This is a valid question.

As a database and security expert, I carefully weigh database changes. However, developers and security zealots typically charge ahead "because compliance."

Email me if you need help with that.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#113

Post by throwawaydba » Sun, Jul 14, 2019, 11:35 AM UTC

zbentley wrote: post #108

zby wrote: post #103

So the article identifies a software bug and a software/config bug as the root cause. That sounds a bit shallow for such a high visibility case - I was expecting something like the https://en.wikipedia.org/wiki/5_Whys method with subplots on why the bugs where not caught in testing. By the way I only clicked on it because I was hoping it would be an occasion to use the methods from http://bayes.cs.ucla.edu/WHY/ - ala…

It is likely that this RCA was shallow because it was intended for everyone--including non-technical users, who (at least in my experience) tend to misinterpret or get confused by deep technical or systemic failure analysis. It would be excellent if Stripe published a truly technical RCA, perhaps for distribution via their tech blog, so that folks like us could get a more complete understanding and what-not-to-do les…

From reading the RCA, this should be the trinity of mysql + orchestrator + vitess. If stripe can't get it right, there is no chance for the others.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#114

Post by mual » Mon, Jul 22, 2019, 4:14 AM UTC

Is this Stripe's first public RCA? Looking through their tweets, there do not appear to be other RCAs for the same "elevated error rates". It seems hard to conclude much from one RCA.