Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

81–90 of 307 posts

Re: The Therac-25 Incident (2021)

#81

Post by grues-dinner »

isopede wrote: post #9

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…

Non-agentic AI is already "killing" people by some definitions. There's a post about someone being talked into suicide on the front page right now, and they are 100% going to get used for something like health insurance and benefits where avoidable death is a very possible outcome. Self-driving cars are also full of "AI" and definitely have killed people already.

Which is not to say that software hasn't killed people before (Horizon, Boeing, probably loads of industrial accidents and indirect process control failures leading to dangerous products, etc, etc). Hell, there's a suspicion that austerity is at least partly predicated on a buggy Excel spreadsheet, and with about 200k excess deaths in a decade (a decade not including Covid) in one country, even a small fraction of those being laid at the door of software is a lot of Theracs.

AI will probably often skate away from responsibility in the same way that Horizon does: by being far enough removed and with enough murky causality that they can say "well, sure, it was a bug, but them killing themselves isn't our fault"

I also find AI copilot things do not work well with embedded software. Again, people YOLOing embedded isn't new, but it might be about to get worse.

Re: The Therac-25 Incident (2021)

#82

Post by jwr »

OskarS wrote: post #77

It's interesting to compare this with the Post Office Scandal in the UK. Very different incidents, but reading this, there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong". For developers, this is a hilariously silly thing, but for non-developers looking at it from the outside, they don't have the capability or training to understand that software can be this f…

No, this is not a "hilariously silly thing" for developers. In fact, I'd say that most developers place way too much trust in software.

I am a developer and whatever software system I touch breaks horribly. When my family wants to use an ATM, they tell me to stand at a distance, so that my aura doesn't break things. This is why I will not get into a self-driving car in the foreseeable future — I think we place far too much confidence in these complex software systems. And yet I see that the overwhelming majority of HN readers are not only happy to be beta-testers for this software as participants in road traffic, but also are happy to get in those cars. They are OK with trusting their life to new, complex, poorly understood and poorly tested software systems, in spite of every other software system breaking and falling apart around them.

[anticipating immediate common responses: 1) yes, I know that self-driving car companies claim that their cars are statistically safer than human drivers, this is beyond the point here. One, they are "safer" largely because they drive so badly that other road participants pay extra attention and accommodate their weirdness, and two, they are still new, complex and poorly understood systems. 2) "you already trust your life to software systems" — again, beyond the point, not quite true as many software systems are built to have human supervision and override capability (think airplanes), and others are built to strict engineering requirements (think brakes in cars) while self-driving cars are not built that way.]

Re: The Therac-25 Incident (2021)

#83

Post by throwaway0261 »

napolux wrote: post #3

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

There was a news story from Norway last year where a car allegedly accelerated by itself, causing the car to fall off the second floor of a parking garage and kill the driver.

Re: The Therac-25 Incident (2021)

#84

Post by mnw21cam »

throwaway0261 wrote: post #80
Forgret wrote: post #52

What surprised me most was that only one developer was working on such an unpredictable technology, whereas I think I need at least 5 developers to be able to discuss options.

One of the benefits of regulations in these areas, is that they require proper tests and documentation. This often requires more than one person to handle the load. We don't want to go back to the 80s YOLO mode just because we need to "move faster". BTW: Relevant XKCD: https://xkcd.com/2347/

Though this XKCD might be even more relevant:

https://xkcd.com/2030/

Re: The Therac-25 Incident (2021)

#85

Post by sgerenser »

kijin wrote: post #30
AdamN wrote: post #17

Earlier quoted context omitted.

Both - and really MCAS was fine but the issue was the metering systems (Pitot tubes) and the handling of conflicting data. That part of the puzzle was definitely a bug in the logic/software.

Remember the Airbus that crashed in the middle of the Atlantic because one of the pilots kept pulling on his yoke, and the computer decided to average his input with normal input from the other pilot? Conflict resolution in redundant systems seems to be one of the weakest spots in modern aircraft software.

Air France 447: https://en.m.wikipedia.org/wiki/Air_France_Flight_447

Inputs were averaged, but supposedly there’s at least a warning: Confused, Bonin exclaimed, "I don't have control of the airplane any more now", and two seconds later, "I don't have control of the airplane at all!"[42] Robert responded to this by saying, "controls to the left", and took over control of the aircraft.[84][44] He pushed his side-stick forward to lower the nose and recover from the stall; however, Bonin was still pulling his side-stick back. The inputs cancelled each other out and triggered an audible "dual input" warning.

Re: The Therac-25 Incident (2021)

#86

Post by mnw21cam »

throwaway0261 wrote: post #83
napolux wrote: post #3

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

There was a news story from Norway last year where a car allegedly accelerated by itself, causing the car to fall off the second floor of a parking garage and kill the driver.

There are plenty of "car allegedly accelerated by itself" incidents, and usually the root cause is the driver mistakenly pressing the accelerator pedal when they think they're pressing the brake pedal. And then swearing blind afterwards that they were braking as hard as they possibly could but the car kept surging forwards.

Re: The Therac-25 Incident (2021)

#87

Post by mnw21cam »

AdamN wrote: post #17
solids wrote: post #13

Earlier quoted context omitted.

Was that a bug or a failure to inform pilots about a new system?

Both - and really MCAS was fine but the issue was the metering systems (Pitot tubes) and the handling of conflicting data. That part of the puzzle was definitely a bug in the logic/software.

It wasn't pitot tubes that had the hardware problem, it was the angle of attack sensor. The software was poorly designed to believe the input from just one fallible angle of attack sensor.

Re: The Therac-25 Incident (2021)

#88

Post by sonicggg »

benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

Not sure why the article is focusing so much on software development. That was just a piece of the problem. The entire product had design flaws. When the FDA for involved, the company wasn't just told to make software updates.

Re: The Therac-25 Incident (2021)

#89

Post by brazzy »

OskarS wrote: post #77

It's interesting to compare this with the Post Office Scandal in the UK. Very different incidents, but reading this, there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong". For developers, this is a hilariously silly thing, but for non-developers looking at it from the outside, they don't have the capability or training to understand that software can be this f…

> there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong"

I think in this case, the thought process was based on the experience with older, electro-mechanical machines where the most common failure modern was parts wearing out.

Since software can, indeed, not "wear out", someone made the assumption that it was therefore inherently more reliable.

Re: The Therac-25 Incident (2021)

#90

Post by varjag »

Fr3dd1 wrote: post #34
AdamN wrote: post #16

Earlier quoted context omitted.

This is true but there also needs to be good developers as well. It can't just be great process and low quality developer practices. There needs to be: 1/ high quality individual processes (development being one of them), 2/ high quality delivery mechanisms, 3/ feedback loops to improve that quality, 4/ out of band mechanisms to inspect and improve the quality.

I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.

My takeaway from observing different teams over years is the talent by a huge margin is the most important component. Throw a team of A performers together and it really doesn't matter what process you make them jump through. This is how a waterfall team got the mankind to the Moon with handwoven core memory but an agile team 10x the size can't fix the software for a family car.
Post reply on HN