Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

71–80 of 307 posts

Re: The Therac-25 Incident (2021)

#71
post #34

Earlier quoted context omitted.

I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.

Right, but if everyone is low quality then there's no one to do that correction. That may seem a bit hypothetical but it can easily happen if you have a company that systematically underpays, which I'm sure many of us don't need to think hard to imagine, in which case they will systematically hire poor developers (because those are the only ones that ever applied).

Sad truth is that average dev is average, but it's not polite to say this out loud. This is particularly important at scale - when you are big tech at some point you hit a wall and no matter how much you pay you can't attract any more good devs, simply because all good devs are already hired. This means that corporate processes must be tailored for average dev, and exceptional devs can only exist in start-ups (or hermetically closed departments). The side effect of that is that whole job market promotes the skill of fitting into corporate environment over the skill of programming. So an a junior dev, for me it makes much more sense to learn how to promote my visibility during useless meetings, rather than learn a new technology. And that's how the bar keeps getting lower.

Re: The Therac-25 Incident (2021)

#72
post #41
post #9

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…

The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

But there is still a difference here. Provenance and proper traceability would have allowed the subpostmasters to show their innocence and prove the system failable.

In the Therac-25 case, the killing was quite immediate and it would have happened even if the correct radiation dose was recorded.

Re: The Therac-25 Incident (2021)

#73
post #61
post #13

Earlier quoted context omitted.

Was that a bug or a failure to inform pilots about a new system?

Not a bug. A non airworthy plane they tried to patch up with software.

The plane was perfectly airworthy without MCAS, that was never the issue. The issue was it handled differently enough at high angles of attack to the 737NG that pilots would've needed additional training or possibly a new type rating without MCAS changing the trim in this situation. The competition (Airbus NEO family) did not need this kind of new training for existing pilots, so airlines being required to do this for new Boeing but not Airbus planes would've been a huge commercial disadvantage.

[edit as I can't reply to the child comment]: The FAA and EASA both looked into the stall characteristics afterwards and concluded that the plane was stable enough to be certified without MCAS and while it did have more of a tenancy to pitch up at high angles of attack it was still an acceptable amount.

Re: The Therac-25 Incident (2021)

#74

Well There's Your Problem podcast, Episode 121: Therac-25 https://www.youtube.com/watch?v=7EQT1gVsE6I

There's also this video from Kyle Hill which is pretty good (I think it's a different incident though, not sure) - https://www.youtube.com/watch?v=Ap0orGCiou8

Re: The Therac-25 Incident (2021)

#75
My "favorite" part:

>One failure occurred when a particular sequence of keystrokes was entered on the VT100 terminal that controlled the PDP-11 computer: If the operator were to press "X" to (erroneously) select 25 MeV photon mode, then use "cursor up" to edit the input to "E" to (correctly) select 25 MeV Electron mode, then "Enter", all within eight seconds of the first keypress and well within the capability of an experienced user of the machine, the edit would not be processed and an overdose could be administered. These edits were not noticed as it would take 8 seconds for startup, so it would go with the default setup

Kinda reminds me how everything is touchscreen nowadays from car interfaces to industry critical software

Re: The Therac-25 Incident (2021)

#76
post #61

Earlier quoted context omitted.

Not a bug. A non airworthy plane they tried to patch up with software.

The plane was perfectly airworthy without MCAS, that was never the issue. The issue was it handled differently enough at high angles of attack to the 737NG that pilots would've needed additional training or possibly a new type rating without MCAS changing the trim in this situation. The competition (Airbus NEO family) did not need this kind of new training for existing pilots, so airlines being required to do this fo…

I may have understood wrong but thought is possible to get into an unrecoverable stall?

Re: The Therac-25 Incident (2021)

#77
It's interesting to compare this with the Post Office Scandal in the UK. Very different incidents, but reading this, there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong". For developers, this is a hilariously silly thing, but for non-developers looking at it from the outside, they don't have the capability or training to understand that software can be this fragile. And they look at a situation like the post office scandal and think "Either this piece of software we paid millions for and was developed by a bunch of highly trained engineers is wrong, or these people are just ripping us off". Same thing with Therac-25, this software had worked on previous models and the rest of the company just had this unspoken assumption that it simply wasn't possible that there was anything wrong with it, so testing it specifically wasn't needed.

Re: The Therac-25 Incident (2021)

#78
post #53
post #9

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…

talk to anyone in the industries about 'automation' on medical or critical infra devices and they will tell you NO. No touching our devices with your rubbish. i am pretty confident they wont let claude touch if it they dont even let deterministic automations run... that being said, maybe there are places. but this is always the sentiment i got. no automating, no scanning, no patching. device is delivered certified an…

> but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon

I don't have the same faith in corporate leadership as you, at least not when they see potentially huge savings by firing some of the expensive developers and using AI to write more of the code.

Re: The Therac-25 Incident (2021)

#79

TIL TheDailyWTF is still active. I'd thought it had settled to greatest hits only some years ago.

This story is kind of old. But also I'm suspicious that this was an AI generated content due to this weird paragraph (one becoming "they"): It's worth noting that there was one developer who wrote all of this code. They left AECL in 1986, and thankfully for them, no one has ever revealed their identity. And while it may be tempting to lay the blame at their feet—they made every technical choice, they coded every bug—…

'They' is a correct singular form for a person of unknown gender. Modern writing overwhelmingly uses it instead of 'he or she', but it has always been correct, has been predominant for a long time, and furthermore it doesn't have anything to do with AI, nor was AI viable as an authoring tool when this article was written, nor is Remy ever going to sell out. What a bizarre comment.

Re: The Therac-25 Incident (2021)

#80
post #52

What surprised me most was that only one developer was working on such an unpredictable technology, whereas I think I need at least 5 developers to be able to discuss options.

One of the benefits of regulations in these areas, is that they require proper tests and documentation. This often requires more than one person to handle the load. We don't want to go back to the 80s YOLO mode just because we need to "move faster".

BTW: Relevant XKCD: https://xkcd.com/2347/

Post reply on HN