Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Root cause analysis: significantly elevated error rates on 2019‑07‑10

stripe.com

Post reply on HN
Print view View original

41–50 of 114 posts

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#41

Post by merlincorey »

bdamm wrote: post #29
ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. When your customers are losing millions of dollars in the minutes you're down, mitigation would be the thing, and analysis can wait. All that is needed is enough forensic data so that testing in earnest to reproduce the condition in the lab can begin. Then get the customers back to…

> In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better.

Bryan Cantrill has a great talk[0] about dealing with fires where he says something to the effect of:

> Now you will find out if you are more operations or development - developers will want to leave things be to gather data and understand, while operations will want to rollback and fix things as quickly as possible

[0] Debugging Under Fire: Keep your Head when Systems have Lost their Mind - Bryan Cantrill: https://www.youtube.com/watch?v=30jNsCVLpAE

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#42

Post by kevinburke »

ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

The odds of you understanding all of the constraints and moving variables in play, and doing situation analysis better than the seasoned ops team at a multibillion dollar company are pretty low. Maybe hold off on the armchair quarterbacking.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#43

Post by uponcoffee »

debt wrote: post #25
cetico wrote: post #12

Earlier quoted context omitted.

Yes this was very surprising. The system was working fine after the cluster restart. There was no need for an emergency rollback. Doing a large rollback based on a hunch seems like an overreaction. It's totally normal for engineers to commit these errors. That's fine. The detail that's missing in this PM is what kind of operational culture, procedures and automation is in place to reduce operator errors. Did the engi…

"That's fine." idk the suits have a very different viewpoint; 30 minutes of downtime for a large financial system isn't fine. it can be very costly.

I think the GP means that as far as incidents occurring, so far as care is (or was) taken to prevent them and learn from them, then that's all one can really reasonably ask for. The first incident falls under that heading and 'is fine' in a 'life happens' sense.

The following incident comes across as reckless and avoidable as there should have been procedures to safely test the rollback (and perhaps there were, but a perfect storm allowed it fail in prod). Lacking details about how the second incident came to be or how they will be prevented going forward places the second incident as 'not fine'.

This information is what the GP comment is asking for.

Compare this PM with Cloudflare's PM, where they detail how they tested rules, what safeguards were in place, how the incident came to be, and how they intend to prevent similar incidents; the impression given here is that they will put up more fire alarms and fire extinguishers but do little fire prevention.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#44

Post by ssalazars »

dps wrote: post #34
ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

(Stripe CTO here) Thanks for the questions. We have testing procedures and deploy mechanisms that enable us to ship hundreds of deploys a week safely, including many which touch our infrastructure. For example, we do a fleetwide version rollout in stages with a blue/green deploy for typical changes. In this case, we identified a specific code path that we believed had a high potential to cause a follow-up incident so…

Thank you for taking the time to respond to my questions. I believe the high potential of causing a follow-up incident was left out of the post (or maybe I missed it?).

I hope that lessons are learned from this operational event, and invest towards building metrics and tooling that allows you to, first of all, prevent issues, and second, shorten the outage/mitigation times in the future.

I'm happy you guys are being open about the issue, and taking feedback from people outside your company. I definitely applaud this.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#45

Post by ssalazars »

bdamm wrote: post #29
ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. When your customers are losing millions of dollars in the minutes you're down, mitigation would be the thing, and analysis can wait. All that is needed is enough forensic data so that testing in earnest to reproduce the condition in the lab can begin. Then get the customers back to…

I understand it. I've worked in AWS, and now in OCI, dealing with systems that affect hundreds-to-thousands of customers, which businesses are at stake.

Mitigation is your top-priority. Bringing the system back to a good shape.

If there needs to be follow-up actions, take the less-impactful steps to prevent another wave.

If there was a deployment, roll-back.

My concern here is, a deployment have been made months ago, and many other changes that could make things worse were introduced. This is the case. The difference between taking an extra 10-20 minutes to make sure everything is fine, versus taking a hot call and causing another outage makes a big difference.

I'm just asking questions based on the documentation provided; I do not have more insights.

I am happy Stripe is being open about the issue, that way many the industry learns and matures regarding software-caused outages. Cloudflare's outage documentation is really good as well.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#46

Post by greenleafjacob »

Silhouette wrote: post #33
greenleafjacob wrote: post #31

Earlier quoted context omitted.

If rollbacks are not safe then you have a change management problem. If you have a good CM system, you should have a timeline of changes that you can correlate against incidents. Most incidents are caused by changes, so you can narrow down most incidents to a handful of changes. Then the question is, if you have a handful of changes that you could roll back, and rollbacks are risk free, then does it make sense to del…

It's not always as simple as that. What if the problem was that something in a change didn't behave as specified and wound up writing important data in an incorrect but retrievable format? Rolling back might not recognise that data properly and could end up either modifying it further so the true data could no longer be retrieved or causing data loss elsewhere as a consequence.

In that case you would probably still roll back to prevent further data corruption and restore the corrupted records from backups.

There are certainly changes that cannot be rolled back such that the affected users are magically fixed, which is not what I am suggesting. In the context of mission critical systems, mitigation is usually strongly preferred. For example, the Google SRE book says the following:

> Your first response in a major outage may be to start troubleshooting and try to find a root cause as quickly as possible. Ignore that instinct!

> Instead, your course of action should be to make the system work as well as it can under the circumstances. This may entail emergency options, such as diverting traffic from a broken cluster to others that are still working, dropping traffic wholesale to prevent a cascading failure, or disabling subsystems to lighten the load. Stopping the bleeding should be your first priority; you aren’t helping your users if the system dies while you’re root-causing. [...] The highest priority is to resolve the issue at hand quickly.”

I have seen too many incidents (one in the last 2 days in fact) that were prolonged because people dismissed blindly rolling back changes, merely because they thought the changes were not the root cause.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#47

Post by scott00 »

sb8244 wrote: post #22
ssalazars wrote: post #19

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was ba…

I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code. However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine…

A rollback without understanding is definitely risky. An uninformed rollback is one of the factors that killed Knight Capital Group in 2012. For those not familiar, the actual problem was they failed to update one of a cluster of eight servers, and the server on the old version was making bad trades. They attempted to mitigate with a rollback, which made all eight servers start to make bad trades. In the end they lost $460 million over the course of about 45 minutes.

The full report is here if you're curious: https://www.sec.gov/litigation/admin/2013/34-70694.pdf

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#48

Post by aeyes »

lethain wrote: post #38
laCour wrote: post #8

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

(Stripe infra lead here) This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)

You might want to clarify this in the post. To me it reads like you knowingly had degraded infra for days leading up to an incident which might have been preventable had you recovered this instances.

Re: Root cause analysis: significantly elevated error rates on 2019‑07‑10

#50

Post by dang »

EugeneOZ wrote: post #24
debt wrote: post #7

"We identified that our rolled-back election protocol interacted poorly with a recently-introduced configuration setting to trigger the second period of degradation." Damn what a mess. Sounds like y'all are rolling out way to many changes too quickly with little to no time for integration testing. It's a somewhat amateur move to assume you can just arbitrarily rollback without consequence, without testing etc. One so…

Not sure why this is downvoted but it all really looks like non-tested deployments to production servers.

Possibly downvoted because of the name-calling ('what a mess', 'amateur move'), which degrades discussion and is against the site guidelines. It's also sort of distasteful to pile on like that.

https://news.ycombinator.com/newsguidelines.html

Post reply on HN