Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Root cause analysis: significantly elevated error rates on 2019‑07‑10

stripe.com

Post reply on HN
Print view View original

31–40 of 114 posts

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#31

Post by greenleafjacob »

sb8244 wrote: post #22
ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code. However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine…

If rollbacks are not safe then you have a change management problem.

If you have a good CM system, you should have a timeline of changes that you can correlate against incidents. Most incidents are caused by changes, so you can narrow down most incidents to a handful of changes.

Then the question is, if you have a handful of changes that you could roll back, and rollbacks are risk free, then does it make sense to delay rolling back any particular change until the root cause is understood?

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#32

Post by davidw »

dps wrote: post #9
chance_state wrote: post #2

This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...

I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.

I don't think either one is particularly "useful" to me as a consumer of the business, other than knowing that "we have top people working on it right now" and there's a plan in place to try and avoid future problems.

What's fun for a software person is that there's a lot of interesting digressions and stuff to learn in the Cloudflare one. The whole explanation of the regexp at the end is something that no one cares about from the business side, but is an interesting read in and of itself.

It's worth noting that yours came out a bit more than a week faster than theirs, which jgrahamc clearly spent a lot of time writing. No idea if anyone cares about the speed with which these things are released...

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#33

Post by Silhouette »

greenleafjacob wrote: post #31
sb8244 wrote: post #22

Earlier quoted context omitted.

I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code. However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine…

If rollbacks are not safe then you have a change management problem. If you have a good CM system, you should have a timeline of changes that you can correlate against incidents. Most incidents are caused by changes, so you can narrow down most incidents to a handful of changes. Then the question is, if you have a handful of changes that you could roll back, and rollbacks are risk free, then does it make sense to del…

It's not always as simple as that. What if the problem was that something in a change didn't behave as specified and wound up writing important data in an incorrect but retrievable format? Rolling back might not recognise that data properly and could end up either modifying it further so the true data could no longer be retrieved or causing data loss elsewhere as a consequence.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#34

Post by dps »

ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

(Stripe CTO here)

Thanks for the questions. We have testing procedures and deploy mechanisms that enable us to ship hundreds of deploys a week safely, including many which touch our infrastructure. For example, we do a fleetwide version rollout in stages with a blue/green deploy for typical changes.

In this case, we identified a specific code path that we believed had a high potential to cause a follow-up incident soon. The course of action was reviewed by several engineers; however we lacked an efficient way to fully validate this change on the order of minutes. We're investing in building tooling to increase robustness in rapid response mechanisms and to help responding engineers understand the potential impact of configuration changes or other remediation efforts they're pushing through an accelerated process.

I think our engineers’ approach was strong here, but our processes could have been better. Our continuing remediation efforts are focused there.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#35

Post by jabart »

dps wrote: post #9
chance_state wrote: post #2

This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage...

I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.

It would be great that in these types of situations if the CC Tokens validity period is extended, or at least known as the documentation states it is short. For our app if the tokens were valid longer, we could write this up as a non-event and retry when things were better.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#36

Post by jdhendrickson »

Silhouette wrote: post #33
greenleafjacob wrote: post #31

Earlier quoted context omitted.

If rollbacks are not safe then you have a change management problem. If you have a good CM system, you should have a timeline of changes that you can correlate against incidents. Most incidents are caused by changes, so you can narrow down most incidents to a handful of changes. Then the question is, if you have a handful of changes that you could roll back, and rollbacks are risk free, then does it make sense to del…

It's not always as simple as that. What if the problem was that something in a change didn't behave as specified and wound up writing important data in an incorrect but retrievable format? Rolling back might not recognise that data properly and could end up either modifying it further so the true data could no longer be retrieved or causing data loss elsewhere as a consequence.

I think the 80 / 20 rule applies here.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#37

Post by sb8244 »

greenleafjacob wrote: post #31
sb8244 wrote: post #22

Earlier quoted context omitted.

I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code. However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine…

If rollbacks are not safe then you have a change management problem. If you have a good CM system, you should have a timeline of changes that you can correlate against incidents. Most incidents are caused by changes, so you can narrow down most incidents to a handful of changes. Then the question is, if you have a handful of changes that you could roll back, and rollbacks are risk free, then does it make sense to del…

Because people make mistakes. Mistakes get fixed in post mortems, retros, best practices, etc. But mistakes will still happen.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#38

Post by lethain »

laCour wrote: post #8

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

(Stripe infra lead here)

This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#39

Post by Silhouette »

jdhendrickson wrote: post #36
Silhouette wrote: post #33

Earlier quoted context omitted.

It's not always as simple as that. What if the problem was that something in a change didn't behave as specified and wound up writing important data in an incorrect but retrievable format? Rolling back might not recognise that data properly and could end up either modifying it further so the true data could no longer be retrieved or causing data loss elsewhere as a consequence.

I think the 80 / 20 rule applies here.

My point was that it's all but impossible for any rollback to be entirely risk-free in this sort of situation. If everything was understood well enough and if everything was working to spec well enough for that to happen, you wouldn't be in a situation where you had to decide whether to make a quick rollback in the first place.

I'm not saying that the decision won't be to do the rollback much of the time. I'm just saying it's unlikely to be entirely without risk and so there is a decision to be considered. Rolling back on autopilot is probably a bad idea no matter how good a change management process you might use, unless perhaps we're talking about some sort of automatic mechanism that could do so almost immediately, before there was enough time for significant amounts of data to be accumulated and then potentially lost by the rollback.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#40

Post by ashelmire »

laCour wrote: post #8

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

If you can think of every possible failure and create monitoring and reporting for it before it happens, then you're the best dev on the planet.
Post reply on HN