Why don't they call 'significantly elevated error rates' an 'outage' instead?
Root cause analysis: significantly elevated error rates on 2019‑07‑10
11–20 of 114 posts
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#12"We identified that our rolled-back election protocol interacted poorly with a recently-introduced configuration setting to trigger the second period of degradation." Damn what a mess. Sounds like y'all are rolling out way to many changes too quickly with little to no time for integration testing. It's a somewhat amateur move to assume you can just arbitrarily rollback without consequence, without testing etc. One so…
Yes this was very surprising. The system was working fine after the cluster restart. There was no need for an emergency rollback.
Doing a large rollback based on a hunch seems like an overreaction.
It's totally normal for engineers to commit these errors. That's fine. The detail that's missing in this PM is what kind of operational culture, procedures and automation is in place to reduce operator errors.
Did the engineer making this decision have access to other team members to review their plan of action? I believe that a group (2-3) of experienced engineers sharing information in real-time and coordinating the response could have reacted better.
Of course, I wasn't there so I could be completely off.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#13Anybody know what database they’re using?
MongoDB is the primary data store used at Stripe.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#14This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...
>This reads like the marketing/PR teams wrote much of it.
The remediation part is quite cautious/generic but overall it seems like a good faith effort by someone constrained by corporate rules.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#15"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.
In this environment:
Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes.
having a few nodes down is perfectly acceptable. I guess they would have had an alert if the number of down nodes exceeded some threshold.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#16"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.
[deleted]
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#17Why don't they call 'significantly elevated error rates' an 'outage' instead?
My guess is that it's because not everything was down so it wasn't a total outage. From the post mortem:
> Stripe splits data by kind into different database clusters and by quantity into different shards.
So in theory any request that didn't interact with the problematic database should have been OK (I don't know if the offending DB was in the critical path of _every_ request).
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#18Why don't they call 'significantly elevated error rates' an 'outage' instead?
Because "A substantial majority of API requests during these windows failed. " implying that there was not a complete outage.
I don't understand why people demand the usage of incorrect language.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#19[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol.
[2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout.
There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was back to normal? How can they decide, and document the change within 20 minutes? Are they using CMs to document changes in production? Were there enough engineers involved in the decision? Clearly all variables were not considered.
To me, this demonstrates poor Operational Excellence values. Your first goal is to mitigate the problem. Then, you need to analyze, understand, and document the root cause. Rolling-back was a poor decision, imo.
Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10
#20"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.
In many HA setups, you're supposed to not have to care if any single thing goes down because it should auto recover
The article said that the node stalled in a way that was unforseen which may have caused standard recovery mechanisms to silently fail.