Live data from Hacker News

Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

blog.danslimmon.com

11–20 of 82 posts

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#11

> When presented with this tradeoff, the path of least resistance is to say “Let’s just keep the threshold lower. We’d rather get woken up when there’s nothing broken than sleep through a real problem.” And I can sympathize with that attitude. Undetected outages are embarrassing and harmful to your reputation. Surely it’s preferable to deal with a few late-night fire drills. > It’s a trap. > In the long run, false po…

I see a lot of this style of writing in articles submitted on HN. I think they are just trying to make the writing more lively, not trying to BS.

A trope of this style is “{interesting half story} but more on that later”.

I don’t think it is a big deal and I don’t see much self promotion here other than vanilla blogging, i.e. sounds like this person is knowledgeable let’s check their bio.

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#12
post #9

I think this article is missing the forest for the trees. The article is about finding the appropriate sensitivity of alerts on some signal in order to maximize the predictive value. But you should care more about the quality of the signals you are monitoring than about the sensitivity of your thresholds. The article mentions load-average as an example signal, but to me, that's a poor signal to monitor. Instead, if y…

> But you should care more about the quality of the signals you are monitoring than about the sensitivity of your thresholds.

This is so true. Case in point: Growatt inverters have - like every other inverter - a maximum voltage on the grid connection at which they will shut down. They're pretty trigger happy about this and fail to take into account the resistance of the feed wire of the inverter to the (much lower impedance) grid hookup. As a result even on cabling sized properly for the interconnect they tend to falsely trigger well before the point where they should. The only way to avoid this problem is to either hack into the inverter somehow (which I've so far failed to do) or to use oversized cables (which isn't always an option).

The sensitivity is fantastic, the quality of the signal is hopeless. Obviously they err on the side of caution but the margin is so ridiculously large that you end up losing a lot of usable power for no reason at all. At least it should allow for either a resistance for the interconnect to be specified so that it can take into account the voltage drop across that wire, which at 10A is appreciable for even short runs of fairly beefy cable.

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#13
post #5

> When presented with this tradeoff, the path of least resistance is to say “Let’s just keep the threshold lower. We’d rather get woken up when there’s nothing broken than sleep through a real problem.” And I can sympathize with that attitude. Undetected outages are embarrassing and harmful to your reputation. Surely it’s preferable to deal with a few late-night fire drills. > It’s a trap. > In the long run, false po…

What do you mean by "this style of writing"? What aspects of the quote do you object to?

At a guess the bit 'let's learn about the base rate fallacy'.

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#14
post #6

I need to sit down and go through the math again, I got lost in the middle somewhere. All I know is our alerts are way too noisy now to the point where they are useless.

Yes! This. This has happened to me at least two previous companies I have worked at. Everybody sets up thresholds on every possible Datadog metric and alerts become useless. That's part of the ethos of monitoring at my current company. We only set up alerts through https://heiioncall.com/ that we are convinced you absolutely need to look at right now. Anything that is not that gets shoved to a slack channel (that I have long since muted).

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#15
post #2

It's a constant pain of mine to try to get people to stop having business as usual or successfully completed $PROCESS emails come out of our batch processes on our teams at work. They absolutely drown my inbox so I'm forced to filter them then the actual failures get buried in the unchecked "batch spam" folders.

My pet peeve is these $PROCESS notifications that go to slack channels. I worked at a company that had an #engineering_humans slack channel because we got chased out of #engineering by bots.

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#16
post #6

I need to sit down and go through the math again, I got lost in the middle somewhere. All I know is our alerts are way too noisy now to the point where they are useless.

I dunno, article doesn't seem to want me to understand. It's just another, "here's a random stats calculation you cant perform in your head, isnt the english a bad way to describe this calculation?!?!? your intuition sucks when i dont explain myself....."

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#17
Now, if you're annoyed by the false positive rate on your actual smoke alarms, go replace the one nearest your kitchen with a photoelectric type, not the standard ionization type that's cheaper, the default style installed, and ought to be illegal in homes (IMO).

There's been quite a bit of research done, generally easy to find if you look, that talks about the difference and tests them, but the short summary:

- Ionization type sensors detect the products of fast flaming combustion and "things cooking in the kitchen." Your oven, if a bit dirty, will reliably trip an ionization type. They are quick on the draw for this. The downside is that they're very, very poor at detecting the sort of slow, smoking, smoldering combustion that is associated with house fires that kill people in the middle of the night.

- The photoelectric type is very good at detecting smoke in the air - but it isn't nearly as prone to false triggers on ovens, a burner burning some spills off, etc.

They've been A/B tested in a wide variety of conditions, and in some cases, the ionization type is a bit quicker. In other cases, the ionization type is slower, by time ranges north of half an hour - I've seen some test reports where there was a 45 minute gap, while the photoelectric type was going off, before the ionization type fired!

In general, "rapid fires during the day" are somewhat destructive to property, but rarely kill people. If your kitchen catches on fire while you're cooking, it may burn the house down, but generally people are able to get out.

The fires that kill people are "slow starting fires during the night" - the sort that smolder for potentially hours, often slowly filling the house with toxic smoke, before actually bursting into open flames. On this sort of fire, the photoelectric type will fire long, long before the ionization type - in some cases, they get around to alarming quite literally "after the occupants are dead from the smoke."

Using smoke alarms as a way to talk about monitoring systems is nice, but in terms of actual smoke detectors, get at least a few photoelectric sorts in the main areas of your home.

Do not get the "combined sensor" sort, since these tend to be and-gated and the worst of both worlds.

Edited to add some resources:

A presentation on the matter from a while back by one of the experts in this field: https://wahigroup.com/Resources/Documents/Ion%20vs%20Photo%2...

Another paper: https://www.semanticscholar.org/paper/Detection-of-Smoke-%3A...

> Full-scale fire tests are carried out to study the effectiveness of the various types of smoke detectors to provide an early warning of a fire. Both optical smoke detectors and ionization smoke detectors have been used. Alarm times are related to human tenability limits for toxic effects, visibility loss and heat stress. During smouldering fires it is only the optical detectors that provide satisfactory safety. With flaming fires the ionization detectors react before the optical ones. If a fire were started by a glowing cigarette, optical detectors are generally recommended. If not, the response time with these two types of detectors are so close that it is only in extreme cases that this difference between optical and ionization detectors would be critical in saving lives.

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#18
post #8

When the oncall gets paged, an SLO should be in jeopardy in a way that requires immediate measures to be taken by a well-trained human as described in actionable terms in a linked playbook. No SLO in jeopardy, or no immediate measure that needs to be taken? Don't page the oncall; send a low-priority ticket for the service owner to investigate the next business day. Steps need to be taken, but they're mechanical in na…

[dead]

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#19
Some complementary reading could be My Philosophy on Alerting ( https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... ) and https://how.complexsystems.fail/

In any case, not all signals are the same. Most systems have a lot of components interacting and what turns to be dangerous is usually a combination of factors, but in the end, what defines that it was or not is that the system is doing what it should. You can put some guessing thresholds, but you must contrast it with that the system works.

And they should be actionable too, at least for alerts instead of slow day notifications, or metrics giving context to perceived problems that could take out the guessing from the thresholds.

Re: Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)

#20
The post is somewhat incomplete without also discussing the cost of the wrong decision.

You obey the smoke alarm because the cost of ignoring the alarm when it is a true positive is potentially infinite (you die). You ignore the car alarm because (1) most likely it is a false positive but also (2) most likely it is somebody else's car.

Post reply on HN