Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Root cause analysis: significantly elevated error rates on 2019‑07‑10

stripe.com

Post reply on HN
Print view View original

21–30 of 114 posts

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#21

Post by Ocha »

dps wrote: post #9
chance_state wrote: post #2

This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...

I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.

For starters maybe provide more details beside the vague information of some feature of some database didnt work as expected. Imagine you are giving this to your employees (especially new ones) to learn something. How much actual useful knowledge is being shared here to learn from?

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#22

Post by sb8244 »

ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code.

However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine that it would most likely be fine. If they did do that, then it's just a mistake and those happen. 20 minutes is enough time to make that call for the team that built it.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#23

Post by chacham15 »

dps wrote: post #9
chance_state wrote: post #2

This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...

I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.

Unexpected things are bound to happen. But, one thing that stuck out to me is that you dont seem to have a safe way to test changes (which would have prevented the second failure). Are there no other environments to test changes on? Is there no way to incrementally roll-out? Is there not another environment which can step in in place of a failing one while you investigate? These seem like fairly common industry practices which help you deal with unexpected failures, but I dont see a mention of if/why these practices failed and if/how that is being remediated.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#24

Post by EugeneOZ »

debt wrote: post #7

"We identified that our rolled-back election protocol interacted poorly with a recently-introduced configuration setting to trigger the second period of degradation." Damn what a mess. Sounds like y'all are rolling out way to many changes too quickly with little to no time for integration testing. It's a somewhat amateur move to assume you can just arbitrarily rollback without consequence, without testing etc. One so…

Not sure why this is downvoted but it all really looks like non-tested deployments to production servers.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#25

Post by debt »

cetico wrote: post #12
debt wrote: post #7

"We identified that our rolled-back election protocol interacted poorly with a recently-introduced configuration setting to trigger the second period of degradation." Damn what a mess. Sounds like y'all are rolling out way to many changes too quickly with little to no time for integration testing. It's a somewhat amateur move to assume you can just arbitrarily rollback without consequence, without testing etc. One so…

Yes this was very surprising. The system was working fine after the cluster restart. There was no need for an emergency rollback. Doing a large rollback based on a hunch seems like an overreaction. It's totally normal for engineers to commit these errors. That's fine. The detail that's missing in this PM is what kind of operational culture, procedures and automation is in place to reduce operator errors. Did the engi…

"That's fine."

idk the suits have a very different viewpoint; 30 minutes of downtime for a large financial system isn't fine. it can be very costly.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#26

Post by [deleted] »

ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

[deleted]

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#27

Post by teraflop »

NikolaeVarius wrote: post #18
jacquesm wrote: post #11

Why don't they call 'significantly elevated error rates' an 'outage' instead?

Because "A substantial majority of API requests during these windows failed. " implying that there was not a complete outage. I don't understand why people demand the usage of incorrect language.

In my mind, a "degradation" would be if some fraction of requests were randomly failing, but they would be likely to eventually succeed if retried. Or if the service itself was essentially accessible, but some non-essential functionality was not working correctly.

On the other hand, if for a significant number of users the site was completely unusable for some period of time, then I think it's fair to use the word "outage". (Even if it's not a complete outage affecting all users.)

I don't know whether other people would interpret these terms the same way I do, nor do I think there's enough information in this blog post to determine for sure which label is more accurate for this particular incident. So personally, I'm not going to be too picky about the wording.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#28

Post by laCour »

NikolaeVarius wrote: post #20
laCour wrote: post #8

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

In many HA setups, you're supposed to not have to care if any single thing goes down because it should auto recover The article said that the node stalled in a way that was unforseen which may have caused standard recovery mechanisms to silently fail.

Right, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#29

Post by bdamm »

ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. When your customers are losing millions of dollars in the minutes you're down, mitigation would be the thing, and analysis can wait. All that is needed is enough forensic data so that testing in earnest to reproduce the condition in the lab can begin. Then get the customers back to working order pronto. 20 minutes seems like a lifetime if in fact they were concerned that the degradation could happen again at any time. 20 minutes seems like just enough time to follow a checklist of actions on capturing environmental conditions, gather a huddle to make a decision, document the change, and execute on it. Commendable actually, if that's what happened.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#30

Post by luizfelberti »

patio11 wrote: post #10
dps wrote: post #9

Earlier quoted context omitted.

I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.

I work at Stripe, on the marketing team, and assisted a bit here. My last major engineering work was writing the backend to a stock exchange. If anyone on HN knows anyone who has the sort of interesting life story where they both know what can cause a cluster election to fail and like writing about that sort of thing, we would eagerly like to make their acquaintance.

Maybe Kyle Kingsbury (aka @aphyr) is the person you are looking for?

https://jepsen.io/services#consulting

Post reply on HN