Root cause analysis: significantly elevated error rates on 2019‑07‑10
1–10 of 114 posts
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#2This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#3Anybody know what database they’re using?
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#4This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...
Nobody from marketing writes this stuff, come on
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#5[deleted]
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#6This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...
Out of curiosity, how would you have preferred to see a shard unable to accept writes? I think in both post-mortems, you would see comparable graphs - usage and then a drop in usage. I think it's easier to document a failed regex versus "here's our cluster architecture that we've been using for 3 months".
Also, does your company's engineering decisions change based on other companies' post-mortems?
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#7"We identified that our rolled-back election protocol interacted poorly with a recently-introduced configuration setting to trigger the second period of degradation."
Damn what a mess. Sounds like y'all are rolling out way to many changes too quickly with little to no time for integration testing.
It's a somewhat amateur move to assume you can just arbitrarily rollback without consequence, without testing etc.
One solution I don't see mentioned, don't upgrade to minor versions ever. And create a dependency matrix so if you do rollback, you rollback all the other things that depend on the thing you're rolling back as well.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#8"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons."
How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#9This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...
I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#10This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...
I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.
I work at Stripe, on the marketing team, and assisted a bit here. My last major engineering work was writing the backend to a stock exchange.
If anyone on HN knows anyone who has the sort of interesting life story where they both know what can cause a cluster election to fail and like writing about that sort of thing, we would eagerly like to make their acquaintance.