Worst Computer Bugs in History: Therac-25 (2017) • Hacker News Board

Re: Worst Computer Bugs in History: Therac-25 (2017)

#71

Post by therein » Sat, Aug 11, 2018, 10:07 PM UTC

ecpottinger wrote: post #60

triska wrote: post #54

Earlier quoted context omitted.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked. Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf : "In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous ma…

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement. When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated. I asked why he does not wire the switches to cut power directly to be on the safe side. His answer "It's to much bo…

In a situation like that, I wouldn't blame him. Consider while building his milling machine how many of these situations he will come across. If he had to make sure there was a hardware failswitch, it would simply not scale.

3D printers are like this too. They have mechanical limit switches [0] that are read only by software. So if there is a bug in the software, nothing is stopping it from pushing the hardware limits and breaking. Same goes the other way around, if this switch is broken, same might happen.

[0] https://i.ebayimg.com/images/g/EYAAAOSwbopZguz4/s-l300.jpg

Re: Worst Computer Bugs in History: Therac-25 (2017)

#72

Post by dev_dull » Sat, Aug 11, 2018, 11:01 PM UTC

> The software consisted of several routines running concurrently. Both the Data Entry and Keyboard Handler routines shared a single variable, which recorded whether the technician had completed entering commands.

Raise your hand if you've made this type of mistake many times in the past. Most of us have the luxury of not having our software bugs affect human lives.

It's a shame they've removed the hardware safety controls. I don't think I'd even feel comfortable programming such a powerful tool without such circuit breakers.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#73

Post by aaron695 » Sat, Aug 11, 2018, 11:02 PM UTC

yc-kraln wrote: post #7

The Therac-25 is part of the core curriculum in computer engineering, but I wonder if it's actually (in the grand scheme of things) hat bad of an incident. Compared with Facebook fomenting ethnic cleansing in Asia, the people who were hurt or died were very limited. Are there any new(er) examples which can show the dangers of a widely distributed, connected horror?

The fact everyone knows it shows how safe computing is!

Except how about the program that allocated shifts to doctors somewhere and needless put someone on who was tired.... so someone dies.

Therac-25's biggest bug that kills is the fact it distracts morons from the real issues.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#74

Post by sverige » Sat, Aug 11, 2018, 11:20 PM UTC

ecpottinger wrote: post #60

triska wrote: post #54

Earlier quoted context omitted.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked. Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf : "In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous ma…

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement. When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated. I asked why he does not wire the switches to cut power directly to be on the safe side. His answer "It's to much bo…

So that's where the youtube videos of CNC milling machine failures come from! TFA noted that major causes of the disaster was the culture and failure to independently unit test the machine.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#75

Post by sverige » Sat, Aug 11, 2018, 11:29 PM UTC

fedorareis wrote: post #24

azernik wrote: post #14

Earlier quoted context omitted.

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

May I point you to the ACM/IEEE-CS Software Engineering Code of Ethics https://ethics.acm.org/code-of-ethics/software-engineering-c... this was a major thing discussed in my professional ethics corse in college.

It's a nice gesture. Unfortunately, it is not required for software engineers to pass any test on this or subscribe to it or otherwise be held to the standard before they are allowed to write and publish software, so unless and until someone forfeits a lot of money and/or their liberty, it will likely not be widely adopted in any practical sense.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#76

Post by dataflow » Sat, Aug 11, 2018, 11:32 PM UTC

Malic wrote: post #35

One of my favorite software horror stories is the one of $32 _billion_ overdraft by the Bank of New York. From "Computer-Related Risks" by Peter G. Neumann, published 1994 (REALLY recommended reading) "One of the most dramatic examples was the $32 billion overdraft experienced by the Bank of New York (BoNY) as the result of the overflow of a 16-bit counter that went unchecked. (Most of the other counters were 32-bits…

Wow. Any idea what would've happened if they hadn't gotten it fixed overnight?

Re: Worst Computer Bugs in History: Therac-25 (2017)

#77

Post by dasil003 » Sat, Aug 11, 2018, 11:46 PM UTC

joering2 wrote: post #64

jobigoud wrote: post #43

Earlier quoted context omitted.

Imaging being called in. Ok guys we don't know what the problem is but it's costing the company $3500 per minute the bug stays unfixed just in interests. No pressure.

Also the follow up question should be ask: was an engineer who stopped the leak and fix the bug awarded to some reasonable point??

No because it was also caused by an engineer, so this could lead to terrifying incentives.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#78

Post by ergothus » Sun, Aug 12, 2018, 12:33 AM UTC

LeoPanthera wrote: post #3

How horrible it must have been for the operator, to realize they had killed two patients, through no fault of their own.

> Due to the frequency at which other malfunctions occurred, and that “treatment pause” typically indicated a low-priority issue, the technician resumed treatment.

I found this the most telling sentence...here the UX was so terrible the operator routinely chose to break the rules... but on some occasions it killed people

Re: Worst Computer Bugs in History: Therac-25 (2017)

#79

Post by sidlls » Sun, Aug 12, 2018, 12:56 AM UTC

dasil003 wrote: post #77

joering2 wrote: post #64

Earlier quoted context omitted.

Also the follow up question should be ask: was an engineer who stopped the leak and fix the bug awarded to some reasonable point??

No because it was also caused by an engineer, so this could lead to terrifying incentives.

How is this different from rewarding a salesman who rescues a sale that a different salesman had botched?

Re: Worst Computer Bugs in History: Therac-25 (2017)

#80

Post by btrettel » Sun, Aug 12, 2018, 1:18 AM UTC

gmueckl wrote: post #68

btrettel wrote: post #63

Earlier quoted context omitted.

Yes, that would be ideal. My impression is that CAD software usually integrates with another software to do stress analysis, e.g., FEA software, so it's more complicated than you've described, but very possible.

How far advanced are methods to automatically setup FEA simulations on arbitrary inputs? The parameter space for FEA methods is pretty large and things like meshing can go terribly wrong and lead to utterly wrong results. Trying to run simulations in the background while a building is being designed seems like a goal to strife for, but to be successful, the software needs to be able to perform equivalently to an expe…

Sorry, I wouldn't really know, as I work in fluid mechanics, not solid mechanics. In fluids I'm getting the impression that automated meshing is becoming fairly robust in some circumstances, to the point where I believe it is sometimes better than an experienced engineer. Solids seems easier in this respect, but as I said, I don't really know.