Worst Computer Bugs in History: Therac-25 (2017) • Hacker News Board

Re: Worst Computer Bugs in History: Therac-25 (2017)

#41

Post by pjmlp » Sat, Aug 11, 2018, 7:04 PM UTC

jacquesm wrote: post #29

pjmlp wrote: post #16

Earlier quoted context omitted.

Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the secur…

It's a huge case of Stockholm Syndrome, end users have been conditioned over years to accept these things as normal and have become engaged in an abusive relationship with their captors who will withhold the little help they are prepared to give if the users dare to complain. No other industry has ever gotten away with this. But with 'software eating the world' change is just around the corner, the first software bug…

I am betting on IoT as the final trigger.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#42

Post by lomnakkus » Sat, Aug 11, 2018, 7:04 PM UTC

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

Damage-by-software usually isn't spectacular (and therefore not likely to get noticed) or not necessarily very directly costly in terms of human lives, but I'd argue it's actually more significant in the long term and in the grand scheme of things. Software rules everything and even slight errors or inefficiencies have absolutely incredible incidental cost.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#43

Post by jobigoud » Sat, Aug 11, 2018, 7:07 PM UTC

Malic wrote: post #35

One of my favorite software horror stories is the one of $32 _billion_ overdraft by the Bank of New York. From "Computer-Related Risks" by Peter G. Neumann, published 1994 (REALLY recommended reading) "One of the most dramatic examples was the $32 billion overdraft experienced by the Bank of New York (BoNY) as the result of the overflow of a 16-bit counter that went unchecked. (Most of the other counters were 32-bits…

Imaging being called in. Ok guys we don't know what the problem is but it's costing the company $3500 per minute the bug stays unfixed just in interests. No pressure.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#44

Post by pjmlp » Sat, Aug 11, 2018, 7:07 PM UTC

anonuser123456 wrote: post #22

pjmlp wrote: post #16

Earlier quoted context omitted.

Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the secur…

Probably not. As a company you just disclaim liability in your terms of service. Jurisdictions that try and override this, simply get excluded from the customer base. The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.

>Probably not. As a company you just disclaim liability in your terms of service.

Thankfully EULAs are void in Europe.

It is all a matter how big the customer base gets, I am hoping eventually we get something like that EU wide.

> The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.

If that was true 1 € shops wouldn't exist, but even those products have more testing than most software out there.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#45

Post by pjmlp » Sat, Aug 11, 2018, 7:08 PM UTC

sgustard wrote: post #33

pjmlp wrote: post #8

Earlier quoted context omitted.

Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.

If your company's motto is "move fast and break things" then you can't punish the developers for bugs.

The company is the one facing the courts not the employees.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#46

Post by oxymoron » Sat, Aug 11, 2018, 7:16 PM UTC

simias wrote: post #39

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

I think that's because for most applications where bodily harm is a possibility you generally (in my experience) have hardware protections that will prevent the software from doing anything stupid. Take an elevator for instance, even if the software controller is bugged (or hacked) and decides that it should drop the cabin from the top floor to the ground level at full speed there are hardware protections (security b…

Your point rings true even in this case. There was another Therac (50? 100? It’s been a while since I read about it) machine which had the same bug, but where noone got hurt due to hardware safeguards.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#47

Post by jacquesm » Sat, Aug 11, 2018, 7:21 PM UTC

pjmlp wrote: post #41

jacquesm wrote: post #29

Earlier quoted context omitted.

It's a huge case of Stockholm Syndrome, end users have been conditioned over years to accept these things as normal and have become engaged in an abusive relationship with their captors who will withhold the little help they are prepared to give if the users dare to complain. No other industry has ever gotten away with this. But with 'software eating the world' change is just around the corner, the first software bug…

I am betting on IoT as the final trigger.

Self driving vehicles, controlled by some griefer would be a pretty harsh demonstration target as well.

There is no way I'll drive an internet connected car, unfortunately I still have to share the road with people that do drive internet connected cars.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#48

Post by AnimalMuppet » Sat, Aug 11, 2018, 7:22 PM UTC

pjmlp wrote: post #8

hedora wrote: post #4

The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.

Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.

I'm pretty sure this incident killed the company. I'm not sure what more "lawful judgment" you want - hold the individual developers legally liable?

Re: Worst Computer Bugs in History: Therac-25 (2017)

#49

Post by AnssiH » Sat, Aug 11, 2018, 7:34 PM UTC

hexane360 wrote: post #34

nv-vn wrote: post #25

Earlier quoted context omitted.

Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if th…

Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/kil…

I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#50

Post by ysleepy » Sat, Aug 11, 2018, 7:42 PM UTC

walrus01 wrote: post #28

azernik wrote: post #14

Earlier quoted context omitted.

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

I am personally more concerned with software engineers and network engineers aiding and abetting the imprisonment, torture and execution of people by repressive regimes, by enabling surveillance technology and fucking with internet traffic analysis. Way more people are going to be hurt in the near term by that than by therac-25 type mistakes. For example if you're a Chinese network engineer, and you can avoid it, don…

You don't have to look that far. The ICE detaining children and violating them are already crimes against humanity. I bet there are IT people working for that agency.

They killed at least one child and are drugging them against their will, while they are forcibly taken from their parents and held in worse conditions than terrorists.