Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Worst Computer Bugs in History: Therac-25 (2017)

blog.bugsnag.com

Post reply on HN
Print view View original

51–60 of 114 posts

Re: Worst Computer Bugs in History: Therac-25 (2017)

#51

Post by umvi »

btrettel wrote: post #11
topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

Seems like now we could have CAD software perform static analysis on the design as well as physics simulation "unit tests" in order to augment testing

Re: Worst Computer Bugs in History: Therac-25 (2017)

#53

Post by dangom »

ysleepy wrote: post #52
dangom wrote: post #2

A descriptive video can be found here: https://www.youtube.com/watch?v=uEvu2PlDhO0

That video is terrible, read the article instead if possible.

Here's the link of (an updated version of) the original accident report: http://sunnyday.mit.edu/papers/therac.pdf

Re: Worst Computer Bugs in History: Therac-25 (2017)

#54

Post by triska »

oxymoron wrote: post #46
simias wrote: post #39

Earlier quoted context omitted.

I think that's because for most applications where bodily harm is a possibility you generally (in my experience) have hardware protections that will prevent the software from doing anything stupid. Take an elevator for instance, even if the software controller is bugged (or hacked) and decides that it should drop the cabin from the top floor to the ground level at full speed there are hardware protections (security b…

Your point rings true even in this case. There was another Therac (50? 100? It’s been a while since I read about it) machine which had the same bug, but where noone got hurt due to hardware safeguards.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked.

Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf:

"In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous machines. The Therac-20 has independent protective circuits for monitoring the electron-beam scanning plus mechanical interlocks for policing the machine and ensuring safe operation. The Therac-25 relies more on software for these functions. AECL took advantage of the computer's abilities to control and monitor the hardware and decided not to duplicate all the existing hardware safety mechanisms and interlocks."

So, regarding these important safety aspects, even the Therac-20 was better than the Therac-25!

The linked post also mentions this:

"Preceding models used separate circuits to monitor radiation intensity, and hardware interlocks to ensure that spreading magnets were correctly positioned."

And indeed, the Therac-20 also had the same software error as the Therac-25! However, quoting again from the paper:

"The software error is just a nuisance on the Therac-20 because this machine has independent hardware protective circuits for monitoring the electron beam scanning. The protective circuits do not allow the beam to turn on, so there is no danger of radiation exposure to a patient."

Re: Worst Computer Bugs in History: Therac-25 (2017)

#55

Post by Jtsummers »

AnssiH wrote: post #49
hexane360 wrote: post #34

Earlier quoted context omitted.

Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/kil…

I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.

The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#56

Post by jsjohnst »

AnssiH wrote: post #49
hexane360 wrote: post #34

Earlier quoted context omitted.

Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/kil…

I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.

Exactly. I don’t blame the technician in the slightest for the race condition, I just called out that ignoring the error state notice was at best careless.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#57

Post by dunmalg »

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

>the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...) from what was also a fairly subtle engineering failure.

It wasn't subtle at all. The entire design was substandard to begin with and didn't meet code. Suspending a walkway from a piece of square tubing made by welding two pieces of C-channel together was ALREADY pants-on-head retarded, and undersized to boot. Deciding it's OK to hang the lower span off the upper span's substandard tubing was just the last step in a long chain of gross engineering negligence.

Calling it a "subtle" failure is like calling Challenger subtle because it was "just a leaky O-ring", or the Apollo I fire subtle because it was "just a tiny spark". There was a completely avoidable cascade of multi-level failure leading up to all of them.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#58

Post by hexane360 »

ghaff wrote: post #37
hexane360 wrote: post #34

Earlier quoted context omitted.

Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/kil…

The pilot example probably isn't the best one.If a plane with that design was nonetheless deemed airworthy by the FAA, pilots would absolutely be expected to know the correct operating procedures for that plane. Look at the "unusual" choice of averaging inputs from the pilot and co-pilot that helped lead to AF 447. It's very reasonable to argue it's bad design, but it was the responsibility of the pilots to know how…

Again, this was a completely undocumented (and unintentional) race condition. There was nothing in the manual stating that "the machine will do bad things if you try to change the mode twice in less than 8 seconds". Averaging of control inputs was a documented, intentional design choice. Being "expected to know the correct operating procedures" doesn't extend to "know how the machine works better than its designers" or "be omniscient".

Re: Worst Computer Bugs in History: Therac-25 (2017)

#59

Post by jsjohnst »

Jtsummers wrote: post #55
AnssiH wrote: post #49

Earlier quoted context omitted.

I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.

The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.

Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either.

I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time, are you completely not responsible if you pull the trigger the sixth time and die?

Re: Worst Computer Bugs in History: Therac-25 (2017)

#60

Post by ecpottinger »

triska wrote: post #54
oxymoron wrote: post #46

Earlier quoted context omitted.

Your point rings true even in this case. There was another Therac (50? 100? It’s been a while since I read about it) machine which had the same bug, but where noone got hurt due to hardware safeguards.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked. Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf : "In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous ma…

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement.

When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated.

I asked why he does not wire the switches to cut power directly to be on the safe side.

His answer "It's to much bother to add the extra circuits."

We are talking less than $20 in parts and a day of his time. If the software fails after sending the controller a message to start moving the head at a certain speed then crashes there is nothing to stop the machine wreaking itself.

E.C.P.

Post reply on HN