Worst Computer Bugs in History: Therac-25 (2017) • Hacker News Board

Re: Worst Computer Bugs in History: Therac-25 (2017)

#31

Post by jacquesm » Sat, Aug 11, 2018, 6:25 PM UTC

anonuser123456 wrote: post #22

pjmlp wrote: post #16

Earlier quoted context omitted.

Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the secur…

Probably not. As a company you just disclaim liability in your terms of service. Jurisdictions that try and override this, simply get excluded from the customer base. The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.

> As a company you just disclaim liability in your terms of service.

Judges might disagree.

> Jurisdictions that try and override this, simply get excluded from the customer base.

Until the customer base is the EU or the US.

> The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.

The market has utterly failed to decide for quality, the market is mostly interested in price and marketing power, quality has never been a very large factor, though in a mature market it might allow some manufacturers to charge a premium for their products.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#32

Post by astraltheman » Sat, Aug 11, 2018, 6:25 PM UTC

fedorareis wrote: post #24

azernik wrote: post #14

Earlier quoted context omitted.

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

May I point you to the ACM/IEEE-CS Software Engineering Code of Ethics https://ethics.acm.org/code-of-ethics/software-engineering-c... this was a major thing discussed in my professional ethics corse in college.

I've never seen this before, this is really interesting. Thank you for the link!

Re: Worst Computer Bugs in History: Therac-25 (2017)

#33

Post by sgustard » Sat, Aug 11, 2018, 6:29 PM UTC

pjmlp wrote: post #8

hedora wrote: post #4

The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.

Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.

If your company's motto is "move fast and break things" then you can't punish the developers for bugs.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#34

Post by hexane360 » Sat, Aug 11, 2018, 6:29 PM UTC

nv-vn wrote: post #25

jsjohnst wrote: post #13

Earlier quoted context omitted.

Honestly, I disagree slightly. Reading the article as well as the original report years ago, I wasn’t left with the feeling the operator made “no fault of their own”. Are they to blame at all, no, but the operator certainly made mistakes. For example, assuming an error is innocuous when you are intentionally delivering radiation to a person is careless at best. Again, the machine is at fault solely, but that doesn’t…

Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if th…

Really?

I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed?

I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...

Re: Worst Computer Bugs in History: Therac-25 (2017)

#35

Post by Malic » Sat, Aug 11, 2018, 6:35 PM UTC

One of my favorite software horror stories is the one of $32 _billion_ overdraft by the Bank of New York.

From "Computer-Related Risks" by Peter G. Neumann, published 1994 (REALLY recommended reading)

"One of the most dramatic examples was the $32 billion overdraft experienced by the Bank of New York (BoNY) as the result of the overflow of a 16-bit counter that went unchecked. (Most of the other counters were 32-bits wide.) BoNY was unable to process the incoming credits from security transfers, while the New York Federal Reserve automatically debited BoNY's cash account. BoNY had to borrow $24 billion to cover itself for 1 day (until the software was fixed), the interest on which was about $5 million. Many customers were also affected by the delayed transaction completions."

Additional reference: https://www.washingtonpost.com/archive/business/1985/12/13/c...

Granted, no one died because of this but ... wow ... that was bad day for some developers somewhere.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#36

Post by ghaff » Sat, Aug 11, 2018, 6:39 PM UTC

azernik wrote: post #14

btrettel wrote: post #11

Earlier quoted context omitted.

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

In addition to the ACM and IEEE, the National Society of Professional Engineers also has a code of ethics. However, relatively few engineers get licensed in the US--it's mostly needed for signing off on drawings for regulators and that sort of thing--and, in fact, the Software Engineering exam is being phased out.

(I took the Engineer-in-Training exam once upon a time but then stropped practicing engineering so I never got the PE.)

Re: Worst Computer Bugs in History: Therac-25 (2017)

#37

Post by ghaff » Sat, Aug 11, 2018, 6:53 PM UTC

hexane360 wrote: post #34

nv-vn wrote: post #25

Earlier quoted context omitted.

Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if th…

Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/kil…

The pilot example probably isn't the best one.If a plane with that design was nonetheless deemed airworthy by the FAA, pilots would absolutely be expected to know the correct operating procedures for that plane.

Look at the "unusual" choice of averaging inputs from the pilot and co-pilot that helped lead to AF 447. It's very reasonable to argue it's bad design, but it was the responsibility of the pilots to know how it worked.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#38

Post by gaius » Sat, Aug 11, 2018, 6:53 PM UTC

azernik wrote: post #14

btrettel wrote: post #11

Earlier quoted context omitted.

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

which software engineers (at least in the US) do not yet have

"It is difficult to get a man to understand something, when his salary depends on his not understanding it." -- Upton Sinclair

Re: Worst Computer Bugs in History: Therac-25 (2017)

#39

Post by simias » Sat, Aug 11, 2018, 6:54 PM UTC

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

I think that's because for most applications where bodily harm is a possibility you generally (in my experience) have hardware protections that will prevent the software from doing anything stupid. Take an elevator for instance, even if the software controller is bugged (or hacked) and decides that it should drop the cabin from the top floor to the ground level at full speed there are hardware protections (security brakes, limitations on the engine itself etc...) that will take over and make sure nobody gets hurt. Therefore in order for something to go completely wrong you need both a software and hardware failure. The main flaw in Therac-25 was arguably that no such protection was present, the hardware should have been designed in order to make the bogus configuration impossible to achieve solely in software.

I think unfortunately this is going to change with the advent of "AI" and related technologies, such as autonomous driving (we've already had a few cases related to self-driving cars after all). When the total enumerable set of possible configurations become too great to exhaustively "whitelist" we won't be able to have foolproof hardware designs anymore. In these situations software bugs can be absolutely devastating.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#40

Post by zzzcpan » Sat, Aug 11, 2018, 7:02 PM UTC

pjmlp wrote: post #16

zzzcpan wrote: post #12

Earlier quoted context omitted.

I don't think it works, at least not within the current legal system. Where it becomes mostly about legal bureaucracy of avoiding responsibility, rather than truly focusing on reliability.

Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the secur…

> a good law firm could probably make something out of it

But that's what I'm saying. If there is a possibility of legal action there will be enough legal bureaucracy to make sure there is something to show in court and avoid responsibility, but not to actually address the problem.