Worst Computer Bugs in History: Therac-25 (2017) • Hacker News Board

Re: Worst Computer Bugs in History: Therac-25 (2017)

#61

Post by hexane360 » Sat, Aug 11, 2018, 8:42 PM UTC

Jtsummers wrote: post #55

AnssiH wrote: post #49

Earlier quoted context omitted.

I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.

The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.

To add to this, the more specific and informative an error message is the more authoritative it appears. "Malfunction 54" is nowhere near as good as "Unable to to set therapy mode"

Re: Worst Computer Bugs in History: Therac-25 (2017)

#62

Post by Jtsummers » Sat, Aug 11, 2018, 8:55 PM UTC

hexane360 wrote: post #61

Jtsummers wrote: post #55

Earlier quoted context omitted.

The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.

To add to this, the more specific and informative an error message is the more authoritative it appears. "Malfunction 54" is nowhere near as good as "Unable to to set therapy mode"

Yep. A good book on this sort of thing is Tragic Design [0]. I was in a pilot "Software Safety" course last year and this book was published just after, but was an excellent companion text for the course. I've been meaning to follow up with them to see what became of that work (the pilot course, my employers at the time were considering making it a mandatory/highly-recommended course for most of their software engineers and designers).

[0] https://www.tragicdesign.com

Re: Worst Computer Bugs in History: Therac-25 (2017)

#63

Post by btrettel » Sat, Aug 11, 2018, 9:00 PM UTC

umvi wrote: post #51

btrettel wrote: post #11

Earlier quoted context omitted.

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

Seems like now we could have CAD software perform static analysis on the design as well as physics simulation "unit tests" in order to augment testing

Yes, that would be ideal. My impression is that CAD software usually integrates with another software to do stress analysis, e.g., FEA software, so it's more complicated than you've described, but very possible.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#64

Post by joering2 » Sat, Aug 11, 2018, 9:10 PM UTC

jobigoud wrote: post #43

Malic wrote: post #35

One of my favorite software horror stories is the one of $32 _billion_ overdraft by the Bank of New York. From "Computer-Related Risks" by Peter G. Neumann, published 1994 (REALLY recommended reading) "One of the most dramatic examples was the $32 billion overdraft experienced by the Bank of New York (BoNY) as the result of the overflow of a 16-bit counter that went unchecked. (Most of the other counters were 32-bits…

Imaging being called in. Ok guys we don't know what the problem is but it's costing the company $3500 per minute the bug stays unfixed just in interests. No pressure.

Also the follow up question should be ask: was an engineer who stopped the leak and fix the bug awarded to some reasonable point??

Re: Worst Computer Bugs in History: Therac-25 (2017)

#65

Post by ghaff » Sat, Aug 11, 2018, 9:11 PM UTC

hexane360 wrote: post #58

ghaff wrote: post #37

Earlier quoted context omitted.

The pilot example probably isn't the best one.If a plane with that design was nonetheless deemed airworthy by the FAA, pilots would absolutely be expected to know the correct operating procedures for that plane. Look at the "unusual" choice of averaging inputs from the pilot and co-pilot that helped lead to AF 447. It's very reasonable to argue it's bad design, but it was the responsibility of the pilots to know how…

Again, this was a completely undocumented (and unintentional) race condition. There was nothing in the manual stating that "the machine will do bad things if you try to change the mode twice in less than 8 seconds". Averaging of control inputs was a documented, intentional design choice. Being "expected to know the correct operating procedures" doesn't extend to "know how the machine works better than its designers"…

Yes. There is a huge difference between undocumented and documented.

And also between pilots flying a plane they are certified on and nurses who are probably less specialized.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#66

Post by mhneu » Sat, Aug 11, 2018, 9:20 PM UTC

simias wrote: post #39

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

I think that's because for most applications where bodily harm is a possibility you generally (in my experience) have hardware protections that will prevent the software from doing anything stupid. Take an elevator for instance, even if the software controller is bugged (or hacked) and decides that it should drop the cabin from the top floor to the ground level at full speed there are hardware protections (security b…

I think unfortunately this is going to change with the advent of "AI" and related technologies, such as autonomous driving

Yes, the potential cost of software bugs is increasing as software does things that no hardware interlock can stop. And worse than that, as a society we largely haven’t realized we need to optimize for worst case (not average) performance of algorithms, because they WILL be attacked. If you’re lucky, they won’t be attacked by sophisticated, well resourced nation-state attackers. But sometimes that will happen.

The rise of complex algorithms to control complex processes is the real difficulty. Facebook’s banning algorithm is an example of something that has been exploited by attackers.

Let’s hope voting software is not the next target where bugs can be exploited. Because changing political decisions can and does produce life-changing effects.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#67

Post by fpgaminer » Sat, Aug 11, 2018, 9:37 PM UTC

yc-kraln wrote: post #7

The Therac-25 is part of the core curriculum in computer engineering, but I wonder if it's actually (in the grand scheme of things) hat bad of an incident. Compared with Facebook fomenting ethnic cleansing in Asia, the people who were hurt or died were very limited. Are there any new(er) examples which can show the dangers of a widely distributed, connected horror?

I'm not familiar with the Facebook incident you are referring to, but it doesn't sound like something caused by a bug? The article is about bugs; i.e. unintentional tragedies. If what you're referring to is not the result of a bug then it's off-topic (though perhaps not unimportant).

Re: Worst Computer Bugs in History: Therac-25 (2017)

#68

Post by gmueckl » Sat, Aug 11, 2018, 9:43 PM UTC

btrettel wrote: post #63

umvi wrote: post #51

Earlier quoted context omitted.

Seems like now we could have CAD software perform static analysis on the design as well as physics simulation "unit tests" in order to augment testing

Yes, that would be ideal. My impression is that CAD software usually integrates with another software to do stress analysis, e.g., FEA software, so it's more complicated than you've described, but very possible.

How far advanced are methods to automatically setup FEA simulations on arbitrary inputs? The parameter space for FEA methods is pretty large and things like meshing can go terribly wrong and lead to utterly wrong results. Trying to run simulations in the background while a building is being designed seems like a goal to strife for, but to be successful, the software needs to be able to perform equivalently to an experienced engineer without guidance. That's a tall order.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#69

Post by Gracana » Sat, Aug 11, 2018, 9:43 PM UTC

ecpottinger wrote: post #60

triska wrote: post #54

Earlier quoted context omitted.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked. Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf : "In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous ma…

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement. When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated. I asked why he does not wire the switches to cut power directly to be on the safe side. His answer "It's to much bo…

Probably not that big a deal. An operator/program can easily damage a CNC mill by jogging a substantial tool (or the spindle itself) into solid material, which is a much more likely scenario.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#70

Post by userbinator » Sat, Aug 11, 2018, 9:46 PM UTC

ecpottinger wrote: post #60

triska wrote: post #54

Earlier quoted context omitted.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked. Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf : "In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous ma…

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement. When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated. I asked why he does not wire the switches to cut power directly to be on the safe side. His answer "It's to much bo…

...but it's not "much bother" to run the control loop through the complexity of software? This seems to be the equivalent of overengineering in software, where something straightforward is instead performed through many layers of abstraction and indirection.

The straightforward way of implementing this with a bidirectional motor is to wire normally-closed limit switches in series with their appropriate direction signals, such that when the switch is actuated it prevents the motor from going in that direction, but it can still move away from the switch.