Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

201–210 of 307 posts

Re: The Therac-25 Incident (2021)

#201

Post by chairmansteve »

benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I would say the real lesson is that the Therac machine should have had hardware interlocks (mentioned but not emphasised in the article).

Re: The Therac-25 Incident (2021)

#202

Post by ramses0 »

haunter wrote: post #75

My "favorite" part: >One failure occurred when a particular sequence of keystrokes was entered on the VT100 terminal that controlled the PDP-11 computer: If the operator were to press "X" to (erroneously) select 25 MeV photon mode, then use "cursor up" to edit the input to "E" to (correctly) select 25 MeV Electron mode, then "Enter", all within eight seconds of the first keypress and well within the capability of an…

   Try quickly typing 1+ 2 + 3 into the iOS 11 Calculator (reddit.com)
   886 points by danso on Oct 24, 2017 | hide | past | favorite | 480 comments
https://news.ycombinator.com/item?id=15538666

...this _exact_ same failure mode in a "less" critical domain (eg: literally your most frequently used "pocket calculator"), unless you're using the calculator for Important Things(tm).

Re: The Therac-25 Incident (2021)

#203

Post by simulator5g »

skinwill wrote: post #64
jacquesm wrote: post #59

Earlier quoted context omitted.

Now we have Chipsoft, arguably one of the worst players in the entire IT space that has a near monopoly (around me, anyway) on IT for hospitals. They charge a fortune, produce crap software and the larger they get the less choice there is for the remainder. It is baffling to me that we should be enabling such hostile players.

Around here we have Epic. If you want a good scare, look up their corporate Willy Wonka-esq jail/campus and their policy of zero remote work.

I've been to that campus. Never worked there but I got a tour when I was like 18 and it seemed like a cool place to work at the time. They clearly took a page from Google's handbook. Its like a giant adult daycare center.

Re: The Therac-25 Incident (2021)

#204

Post by codeulike »

hahn-kev wrote: post #49
SCdF wrote: post #41

Earlier quoted context omitted.

The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.

It was a distributed system lashed together by 'consultants' (read: recent graduates with little real world software engineering experience) in an era where best practices around distributed systems were non-existent. They weren't even thinking about what kind of data inconsistencies they might end up with.

Re: The Therac-25 Incident (2021)

#205

Post by simulator5g »

V__ wrote: post #94
hahn-kev wrote: post #49

Earlier quoted context omitted.

From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.

> Engineers are legally obligated to report unsafe conduct, activities or behaviours of others that could pose a risk to the public or the environment. [1] If software "engineers" want to be taken seriously, then they should also have the obligation to report unsafe/broken software and refuse to ship unsafe/broken software. The developers are just as much to blame as the post office: > Fujitsu was aware that Horizon…

I don't think it's fair to blame individual developers for a systemic failure. Its not their fault there is no governing body to award or remove the title of "software engineer" and promote the concept of a software engineer refusing to do something without harming their career. Other engineering disciplines have laws, lobbied for by their governing body, that protect the ability of individual engineers to prevent higher-ups from making grave mistakes.

Re: The Therac-25 Incident (2021)

#206

Post by dezgeg »

zackmorris wrote: post #184

Our power went off a couple off weeks ago due to wind probably knocking a branch into a power line. Now our Frigidaire microwave runs with the door open. Supposedly there are mechanical switches that prevent that, but evidently "modern" microwaves can control the gun through the logic board. The engineering failures that led to this, from conceptual to design to internal control, boggle my mind. I'm not even sure whe…

It's possible that it's just operating the lights and turntable without actually cooking - if you search HN you find that failure mode.

Re: The Therac-25 Incident (2021)

#207

Post by kulahan »

franktankbank wrote: post #127
ChrisMarshallNY wrote: post #120

Earlier quoted context omitted.

I worked for a company that manufactured some of the highest-Quality photographic and scientific equipment that you can buy. It was expensive as hell, but our customers seemed to think it was worth it. > It's the end result of a process In my experience, it's even more than that. It's a culture .

A culture of high-quality engineering, no doubt. Made up of: high quality engineers!

Unfortunately, software developers are the absolute most offensive use of the word "engineer", because 99.9% of the stuff this field makes is a competition to take the most unique approach to a solution, then getting it bandaged together with gum and paperclips.

If this industry wants to be respected, it should start trying to be actual engineers. There should be tons and tons of standards which are enforced legally, but this is not often the case. Imagine if there were no real legal guardrails in, say, bridge building!

edit: and imagine if any time you brought up this issue, bridge builders cockily responded with "well stuff seems to work fine so..."

Re: The Therac-25 Incident (2021)

#208

Post by WalterBright »

benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I'm going to disagree.

I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident.

The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case."

For the Therac-25, that means a detector of the amount of radiation being generated, which will cut it off if it exceeds a safe value. I'd also add that the radiation generator be physically incapable of generating excessive radiation.

Re: The Therac-25 Incident (2021)

#209

Post by WalterBright »

bell-cot wrote: post #101
vorgol wrote: post #39

Earlier quoted context omitted.

I was going to recommend that exact podcast episode but you beat me to it. Totally worth listening, especially if you're interested in software bugs. Another interesting fact mentioned in the podcast is that the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. Excellent demonstration of the Swiss Cheese Model: https:…

>> the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed. > the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. #1 virtue of electromechanical failsafes is that their conception, design, implementation, and failure mode…

> And PHB's all know that picking up nickles in front of the steamroller is how you get to the C-suite.

Blaming it on PHB's is a mistake. There were no engineering classes in my degree program about failsafe design. I've known too many engineers who were insulted by my insinuations that their design had unacceptable failure modes. They thought they could write software that couldn't possibly fail. They'd also tell me that they could safely recover and continue executing a crashed program.

This is why I never, ever trust software switches to disable a microphone, software switches that disable disk writes, etc. The world is full of software bugs that enable overriding of their soft protections.

BTW, this is why airliners, despite their advanced computerized cockpit, still have an old fashioned turn-and-bank indicator that is independent of all that software.

Re: The Therac-25 Incident (2021)

#210

Post by kccqzy »

wat10000 wrote: post #112
0xDEAFBEAD wrote: post #98

Earlier quoted context omitted.

Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".

The lesson is not to write a mess like that. It might seem obvious, but it has to be learned.

Software engineering has advanced in the past few decades that the kind of code considered a "mess" has expanded.
Post reply on HN