Earlier quoted context omitted.
> But if I told someone that I don't need to know the effects of v7.1.3 before patching it because I can't think of a mechanism for it going wrong, and besides, we already released all those other versions and they were fine, I'd hopefully be fired. But this isn't what they're saying. They're saying that there are few to no ways for it to not have any detectable issues for 6 months and then suddenly go wrong (which I…
> But this isn't what they're saying. They're saying that there are few to no ways for it to not have any detectable issues for 6 months and then suddenly go wrong (which I'll note is also true for software!). Over the years I've become a little obsessed with imagining the ways things can fail (I'm just as fun at parties as you can probably imagine). An example that immediately jumps to mind where that is not true fo…
Some engineer creates a fix by turning off some recently added experiment responsible for the service degradation. The rollback passes all unit tests and integration tests. You know this fix is low-risk (maybe because your service was healthy for the ~1 year prior to the rollout). Even if there is some risk that this rollback breaks the service at some point in the future, in comparison to the known revenue loss the expected loss of that breakage would be minimal.
Another engineer objects to the rollback saying that "they don't know of the long term effects of such a rollback." You try to counter their claims by showing that previous rollbacks were almost always successful, and when they caused another failure that failure was not nearly as bad as the original service degradation. Also, the rollback passes all tests. That engineer then repeats "we don't know of the long term effects of this particular rollback, and I'm skeptical that the fix is good because the failing experiment and rollback was identified so quickly."
(Rollback vs. fix-forward doesn't actually matter; the point is that all available knowledge shows that the fix is low risk, historically supported, and that there is likely no mechanism for random failure in 6+ months.)
Do you think it would be correct to prefer the second engineer's claims?