Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
1–10 of 82 posts
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#2It's a constant pain of mine to try to get people to stop having business as usual or successfully completed $PROCESS emails come out of our batch processes on our teams at work. They absolutely drown my inbox so I'm forced to filter them then the actual failures get buried in the unchecked "batch spam" folders.
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#3It's a constant pain of mine to try to get people to stop having business as usual or successfully completed $PROCESS emails come out of our batch processes on our teams at work. They absolutely drown my inbox so I'm forced to filter them then the actual failures get buried in the unchecked "batch spam" folders.
The one that pushes buttons is the alarms that have no docs attached so when they blow off at 2AM, they just get muted until someone comes in and complains at 6AM.
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#4> When presented with this tradeoff, the path of least resistance is to say “Let’s just keep the threshold lower. We’d rather get woken up when there’s nothing broken than sleep through a real problem.” And I can sympathize with that attitude. Undetected outages are embarrassing and harmful to your reputation. Surely it’s preferable to deal with a few late-night fire drills.
> It’s a trap.
> In the long run, false positives can — and will often — hurt you more than false negatives. Let’s learn about the base rate fallacy.
Not sure about anyone else, but speaking of alarms, this style of writing trips my "self-promoting snake-oil Internet bullshitter" alarm. It's like nails on a damn chalkboard, and if you're writing like this, you've already lost me; however, maybe I ought not be pointing that out, since signals are nice to have.
Incidentally, I wasn't sure which way the author was gonna go with the core analogy. My smoke alarms have false-alarmed probably 10x as much as my car alarm, even counting times one of us has hit the alarm button on the fob by accident. I've certainly never been so annoyed by my car alarm that I've ripped it out and stuck it in a freezer, as I have with a smoke alarm.
(If I were writing like the author I suppose that last part would have read:
"I've certainly never been so annoyed by my car alarm that I've ripped it out and stuck it in a chest freezer.
I have, with a smoke alarm."
Except also I'd have found a way to use "we" and "you" a bunch.)
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#5> When presented with this tradeoff, the path of least resistance is to say “Let’s just keep the threshold lower. We’d rather get woken up when there’s nothing broken than sleep through a real problem.” And I can sympathize with that attitude. Undetected outages are embarrassing and harmful to your reputation. Surely it’s preferable to deal with a few late-night fire drills. > It’s a trap. > In the long run, false po…
What do you mean by "this style of writing"? What aspects of the quote do you object to?
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#6I need to sit down and go through the math again, I got lost in the middle somewhere. All I know is our alerts are way too noisy now to the point where they are useless.
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#7I do like how the author presents the case for how damaging false-positives can be in SRE monitoring. But, FYI, it can get worse if these monitors are hooked to self-actuating feedback loops! I recently wrote about a production incident on the Heii On-Call blog, in the context of witnessing how Kubernetes liveness probes and CPU limits worked together to create a self-reinforcing CrashLoopBackOff. [1] Partially because the liveness probe thresholds (timeoutSeconds and failureThreshold fields) were too aggressive.
We have a similar message about setting monitoring thresholds in our documentation [2] because users have to explicitly specify a downtime timeout before they’re alerted about their website / API endpoint / cron job being down. The timeout / "grace period" is necessary because in many cases a failure is some transient network glitch which will fix itself before a human is alerted.
If you make the timeout too short, you’ll get lots of false positive alerts, and as the article says, your on-call engineers will be overwhelmed or just start ignoring the alerts.
If you make the timeout too long, it just takes that many minutes of downtime longer before you find out about it.
It may sound counterintuitive, but the latter is usually preferable. :)
[1] https://heiioncall.com/blog/kubernetes-liveness-probes-and-c...
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#8When the oncall gets paged, an SLO should be in jeopardy in a way that requires immediate measures to be taken by a well-trained human as described in actionable terms in a linked playbook.
No SLO in jeopardy, or no immediate measure that needs to be taken? Don't page the oncall; send a low-priority ticket for the service owner to investigate the next business day.
Steps need to be taken, but they're mechanical in nature or otherwise don't give the SRE an opportunity to exercise their brain in an interesting fashion? Replace the alert with an automated handler that only pages the oncall if it encounters an exception.
No playbook, or the playbook consists of useless non-actionable items like, "This alert means the service is running out of frobs"? Write a playbook that explains what the oncall is expected to do when the service needs frobs.
Edit: A dead reply asks if I've ever experienced a novel incident. Of course. Say, for instance, a "This should never happen" error-level log is suddenly happening like crazy, for the first time ever. In that case, you page the oncall, they do their best to debug it, see if they can reach the SWE service owners, read through the code to see if it could be an indicator that SLOs are being violated (e.g., user data corruption) or might be violated soon, and then write a stub playbook to be fleshed out the next business day, probably alongside a code change to handle this situation without spamming the logs so much.
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#9I think this article is missing the forest for the trees.
The article is about finding the appropriate sensitivity of alerts on some signal in order to maximize the predictive value.
But you should care more about the quality of the signals you are monitoring than about the sensitivity of your thresholds.
The article mentions load-average as an example signal, but to me, that's a poor signal to monitor. Instead, if your SLO is defined for error rate, alert on error rate.
Alerts on your SLO will have a high predictive value for predicting violations of your SLO, by definition. The tunable parameter here is the time window, not the threshold. E.g. if your error budget is defined for a 30d window, you may want alerts at the SLO threshold for 24h and 1h windows.
Alert on causes, not symptoms.
Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
#10I need to sit down and go through the math again, I got lost in the middle somewhere. All I know is our alerts are way too noisy now to the point where they are useless.
Alert fatigue... it's common when alerts are non-actionable and it causes a lot of downtime.