Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

181–190 of 307 posts

Re: The Therac-25 Incident (2021)

#181

Post by ndsipa_pomu »

OskarS wrote: post #77

It's interesting to compare this with the Post Office Scandal in the UK. Very different incidents, but reading this, there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong". For developers, this is a hilariously silly thing, but for non-developers looking at it from the outside, they don't have the capability or training to understand that software can be this f…

I'd consider the Post Office Scandal to be far more malicious. The higher ups in the post office were getting bonuses IIRC according to how much money was "recovered" (defrauded) from the subpostmasters. Also there was a lot of lying to the courts and ministers about the reliability of the software.

As far as I know, the Therac-25 incidents were reasonably honest mistakes.

Re: The Therac-25 Incident (2021)

#182

Post by mrguyorama »

michaelt wrote: post #6

I'd be interested in knowing how many of y'all are being taught about this sort of thing in college ethics/safety/reliability classes. I was taught about this in engineering school, as part of a general engineering course also covering things like bathtub reliability curves and how to calculate the number of redundant cooling pumps a nuclear power plant needs. But it's a long time since I was in college. Is this sort…

The therac-25 was just one of the many incidents we covered in my Software Ethics course for my Computer Science degree. The problem is not "we have to teach it", the problem is that at least half the talented people in the room with me in that class considered the entire thing "a joke" bullshit class that just wasted their time.

You can't teach people to care.

Re: The Therac-25 Incident (2021)

#183

Post by OskarS »

ndsipa_pomu wrote: post #181
OskarS wrote: post #77

It's interesting to compare this with the Post Office Scandal in the UK. Very different incidents, but reading this, there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong". For developers, this is a hilariously silly thing, but for non-developers looking at it from the outside, they don't have the capability or training to understand that software can be this f…

I'd consider the Post Office Scandal to be far more malicious. The higher ups in the post office were getting bonuses IIRC according to how much money was "recovered" (defrauded) from the subpostmasters. Also there was a lot of lying to the courts and ministers about the reliability of the software. As far as I know, the Therac-25 incidents were reasonably honest mistakes.

I agree, that is very true, Therac-25 was incompetence, Post Office was incompetence with a heavy dose of malice. This aspect just steuck me as similar, the unquestioning belief in the infallibility of software.

Re: The Therac-25 Incident (2021)

#184

Post by zackmorris »

Our power went off a couple off weeks ago due to wind probably knocking a branch into a power line. Now our Frigidaire microwave runs with the door open.

Supposedly there are mechanical switches that prevent that, but evidently "modern" microwaves can control the gun through the logic board.

The engineering failures that led to this, from conceptual to design to internal control, boggle my mind. I'm not even sure where to send a complaint or if it would result in any kind of compensation. Because billion dollar corporations know that they'll never have to face any kind of corporate death penalty because they're protected by limited liability. So we'll just buy another $150 microwave instead.

Are smaller companies better at engineering safety? Evidently not.

Re: The Therac-25 Incident (2021)

#185

Post by InvisibleUp »

mlnhd wrote: post #174
michaelt wrote: post #6

I'd be interested in knowing how many of y'all are being taught about this sort of thing in college ethics/safety/reliability classes. I was taught about this in engineering school, as part of a general engineering course also covering things like bathtub reliability curves and how to calculate the number of redundant cooling pumps a nuclear power plant needs. But it's a long time since I was in college. Is this sort…

This and Tacoma Narrows are literally the only topics covered in engineering ethics, which itself is literally only a one hour presentation.

Don’t forget the Hyatt Regency walkway, too.

Re: The Therac-25 Incident (2021)

#186

Post by throwawayoldie »

isopede wrote: post #9

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…

They killed "only" about 350 people combined, but the two fatal crashes of the Boeing 737 MAX in 2018 and 2019 were due to poor quality software:

https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Au...

Re: The Therac-25 Incident (2021)

#187

Post by ipython »

vorgol wrote: post #39
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I was going to recommend that exact podcast episode but you beat me to it. Totally worth listening, especially if you're interested in software bugs. Another interesting fact mentioned in the podcast is that the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. Excellent demonstration of the Swiss Cheese Model: https:…

Don’t worry we are poised to re learn all these lessons once again with our fancy new agentic generative ai systems.

The mechanical interlock essentially functioned as a limit outside of the control system. So you should build an ai system the same way- enforcing restrictions on the security agency from outside the control of the ai itself. Of course that doesn’t happen and devs naively trust that the ai can make its own security decisions.

Another lesson from that era we are re learning- in-band signaling. Our 2025 version of the “blue box” is in full swing. Prompt injection is just a side effect of the fact that there is no out of band instruction mechanism for llms.

Good news is - it’s not hard to learn the new technology when it’s just a matter of rediscovering the same security issues with a new name!

Re: The Therac-25 Incident (2021)

#188

Post by bongodongobob »

skinwill wrote: post #64
jacquesm wrote: post #59

Earlier quoted context omitted.

Now we have Chipsoft, arguably one of the worst players in the entire IT space that has a near monopoly (around me, anyway) on IT for hospitals. They charge a fortune, produce crap software and the larger they get the less choice there is for the remainder. It is baffling to me that we should be enabling such hostile players.

Around here we have Epic. If you want a good scare, look up their corporate Willy Wonka-esq jail/campus and their policy of zero remote work.

Their hiring is insane too. Years ago I applied for a Sys admin I position. After 3 rounds of interviews, they gave me a test to take. Except it was something you'd give a senior dev. They specced out a fake language, super esoteric too, it was like a combination of brainfuck and assembly, and asked me to solve problems with it, and debug code written in it. I could have toughed my way through it, but I was so dissatisfied with how much of my time they'd already wasted that I decided I didn't want to work there.

Re: The Therac-25 Incident (2021)

#189

Post by bongodongobob »

zackmorris wrote: post #184

Our power went off a couple off weeks ago due to wind probably knocking a branch into a power line. Now our Frigidaire microwave runs with the door open. Supposedly there are mechanical switches that prevent that, but evidently "modern" microwaves can control the gun through the logic board. The engineering failures that led to this, from conceptual to design to internal control, boggle my mind. I'm not even sure whe…

I have a microwave from the early 80s. If I stick a pencil in the door latch I can get it to run with the door open as well. It's not the demon core. Just don't stick your head in it when it's running.

Re: The Therac-25 Incident (2021)

#190

Post by hinkley »

0xDEAFBEAD wrote: post #98
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".

I bring up Knight Capital every time people start acting like feature toggles will solve every problem we have with feature rollout.

KC lost over $400 million in less than an hour due to an old feature toggle and a problem with their deployment process.

Post reply on HN