Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Worst Computer Bugs in History: Therac-25 (2017)

blog.bugsnag.com

Post reply on HN
Print view View original

91–100 of 114 posts

Re: Worst Computer Bugs in History: Therac-25 (2017)

#91

Post by dasil003 »

sidlls wrote: post #87
dasil003 wrote: post #81

Earlier quoted context omitted.

Because you can't purposefully botch a sale in order to later recover it. Also because you can't avoid botching sales by being more conservative or adding more process. In short, sales and engineering have basically nothing in common.

I don't think GP was talking about a scenario where the same engineer who created the bug fixed it and gets rewarded, rather one where a different engineer fixes it. Of course it wouldn't make sense as you describe it.

How do you decide objectively who is responsible for every single bug? The whole thing is ripe for abuse from all sides. You need a blameless culture to have good engineering, not a bounty-based one.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#92

Post by matheusmoreira »

> users will ignore cryptic error messages, particularly if they occur often

It's not just cryptic error messages. Pretty much anything that requires the attention of people will end up being ignored eventually. For example:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4894506/

I've also read about an anesthesiologist who turned off the alarms because they annoyed him. One day he failed to secure the endotracheal tube during a surgery, it came off and nobody noticed. The result was cardiac arrest, brain damage, multiple organ failure, sepsis and death.

Monitoring hardware is very sensitive so it will fire off alarms if anything changes, no matter how small. The more sensitive a test is, the more false positives you get. This is extremely demanding of a health care professional's attention, which in practice is multiplexed between countless patients.

Up to 99% of these alarms and messages will do nothing but get in the way of people. These represent false positives, disconnected cables, and other minor failures that don't represent a real danger and can be easily fixed. People will get used to the alarms, and will learn to ignore them.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#93

Post by matheusmoreira »

jacquesm wrote: post #21
classichasclass wrote: post #5

Therac is one of the reasons I get nervous about "health hacking." Yes, people can verifiably benefit from some of the advancements made in this movement, like the DIY diabetic insulin pump, and yes, I prefer to see such advancements be open source than locked up in proprietary designs and trade secrets. And there probably is room in health regulation for trimming the red tape anyway even for innovations originating…

I posted a link about a insulin pump that could be hacked into remotely. Even if it isn't 'health hacking' because you've bought a product from what you thought was a reputable vendor there are no guarantees that it will be secure and bug free.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3262727/

> An unauthorized third party can interfere with pump communication and undermine patient safety

> we confirmed this through laboratory experiments by sending commands to an insulin pump using an unauthorized remote programmer at a distance of 100 ft

> Thus, the specifically identified issues are a security breach that could result in:

> (1) changing already-issued wireless pump commands;

> (2) generating unauthorized wireless pump commands;

> (3) remotely changing the software or settings on the device;

> (4) denying communication with the pump device.

People can also attack the blood glucose monitors and the data they report to the pump system.

Scary.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#94

Post by jibal »

In 1973 I programmed for the oncology department at L.A. County/US Medical Center. We had a Varian Clinac linear accelerator with computer-readable and -drivable motors. The clinicians would manually position the Clinac (with the patient on it) for the first treatment, and the position would be saved in the patient's computerized file and restored on subsequent treatments.

For some treatments, a metal wedge would be placed within the beam to attenuate it more at the thick end of the wedge. Because of the non-linear attenuation along the length of the physical metal wedge, dosages were difficult to calculate.

Someone got the bright idea of creating a software wedge by slowly moving the treatment couch at the same time as closing the beam aperture, so that there would be 100% exposure at one end of the "wedge" and 0% at the other, with a linearly decreasing distribution across the whole wedge.

I was the programmer for this project, and we had just started testing it with a sheet of X-ray film on the couch when I received an offer I couldn't refuse to go work elsewhere.

I'm glad that I departed before they started using this on live patients.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#95

Post by jibal »

oxymoron wrote: post #46
simias wrote: post #39

Earlier quoted context omitted.

I think that's because for most applications where bodily harm is a possibility you generally (in my experience) have hardware protections that will prevent the software from doing anything stupid. Take an elevator for instance, even if the software controller is bugged (or hacked) and decides that it should drop the cabin from the top floor to the ground level at full speed there are hardware protections (security b…

Your point rings true even in this case. There was another Therac (50? 100? It’s been a while since I read about it) machine which had the same bug, but where noone got hurt due to hardware safeguards.

> It’s been a while since I read about it

It's in the article you're commenting on.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#96

Post by jibal »

matheusmoreira wrote: post #92

> users will ignore cryptic error messages, particularly if they occur often It's not just cryptic error messages. Pretty much anything that requires the attention of people will end up being ignored eventually. For example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4894506/ I've also read about an anesthesiologist who turned off the alarms because they annoyed him. One day he failed to secure the endotracheal tub…

> Pretty much anything that requires the attention of people will end up being ignored eventually.

Perhaps, but your example is not supporting evidence. The PACU alarms were muted, precisely because they were so hard to ignore.

> Monitoring hardware is very sensitive so it will fire off alarms if anything changes, no matter how small. The more sensitive a test is, the more false positives you get.

This is fixable. The problem is the same as the one in the Therac-25 case ... severe and inconsequential alerts are indistinguishable.

Here's an particularly enjoyable piece of literature crafted around an instance of alarm fatigue: https://gutenberg.ca/ebooks/smithcordwainer-deadladyofclownt...

Re: Worst Computer Bugs in History: Therac-25 (2017)

#97

Post by jibal »

joering2 wrote: post #64
jobigoud wrote: post #43

Earlier quoted context omitted.

Imaging being called in. Ok guys we don't know what the problem is but it's costing the company $3500 per minute the bug stays unfixed just in interests. No pressure.

Also the follow up question should be ask: was an engineer who stopped the leak and fix the bug awarded to some reasonable point??

Why should they be? The difficulty and quality of the work isn't dependent on the severity of the bug. Basically you would be creating a lottery.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#98

Post by iforgotpassword »

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

The difference here is that you treat one person at a time, and the bug doesn't get triggered every time. A bridge just happens to be used by a lot of people at the same time. The bridge didn't collapse three times until people figured out something was wrong.

This applies to the vast majority of fields where software is in use, except maybe planes, trains and nuclear power plants (where I sure as hell hope are a bunch of hardware safeguards in place). So in a sense, we software developers just got lucky here that our mistakes only kill one or very few at a time in most use cases (if at all).

It's still insane how according to the article they apparently just had that thing developed using emulated hardware with no proper security audits, safety guidelines or formal verification.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#99

Post by perl4ever »

jsjohnst wrote: post #59
Jtsummers wrote: post #55

Earlier quoted context omitted.

The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.

Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either. I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “…

"if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time"

I'm not sure that's a valid analogy. Isn't it the case rather that the revolver said "that's bad" but a person said "it's fine, ignore that, it always does that"? Which, in my experience, happens all the time in the workplace. If you didn't trust other people you work with, it would defeat the purpose of being in an organization at all.

Not to mention, if pulling the trigger repeatedly was required for normal operations, you really can't blame the operator at all.

Post reply on HN