In my university this case was (and probably still is) subject of the first lecture in the first semester. A lot to learn here and one of the prime examples how the DEPOSE model [Perrow 1984] works for software engineering.
The Therac-25 Incident (2021)
51–60 of 307 posts
Re: The Therac-25 Incident (2021)
#52What surprised me most was that only one developer was working on such an unpredictable technology, whereas I think I need at least 5 developers to be able to discuss options.
Re: The Therac-25 Incident (2021)
#53I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…
talk to anyone in the industries about 'automation' on medical or critical infra devices and they will tell you NO. No touching our devices with your rubbish.
i am pretty confident they wont let claude touch if it they dont even let deterministic automations run...
that being said, maybe there are places. but this is always the sentiment i got. no automating, no scanning, no patching. device is delivered certified and any modifications will invalidate that. any changes need to be validated and certified.
its a different world that makin apps thats for sure.
not to say mistakes arent made and change doesnt happen, but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon... give the folks in safety critical system engineering some credit..
Re: The Therac-25 Incident (2021)
#54Wondering if that "one developer" is here on HN.
Hahaha, it would be interesting, maybe he just commented on the post here?
Re: The Therac-25 Incident (2021)
#55Earlier quoted context omitted.
The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.
From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.
as you point out this was a messup on a lot of levels. its an interesting effect tho not to be dismissed. how your software works and how its perceived and trusted can impact people psychologically.
Re: The Therac-25 Incident (2021)
#56Earlier quoted context omitted.
The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.
From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.
Given whole truth testimony?
Re: The Therac-25 Incident (2021)
#57Earlier quoted context omitted.
Was that a bug or a failure to inform pilots about a new system?
Both - and really MCAS was fine but the issue was the metering systems (Pitot tubes) and the handling of conflicting data. That part of the puzzle was definitely a bug in the logic/software.
That wasn't a bug.
They deliberately designed it to only look at one of the Pitot tubes, because if they had designed it to look at both, then they would have had to implement a warning message for conflicting data.
And if they had implemented a warning message, they would have had to tell the pilots about the new system, and train them how to deal with it.
It wasn't a mistake in logic either. This design went through their internal safety certification, and passed.
As far as I'm aware, MCAS functioned exactly as designed, zero bugs. It's just that the design was very bad.
Re: The Therac-25 Incident (2021)
#58Some sanity checks are always a good idea before running such destructive action (IF beam_strength > REASONABLY_HIGH_NUMBER THEN error). Of course the UI bug is hard to catch, but the sanity check would have prevented this completely and the machine would just end up in an error, rather than killing patients.
invariants are so useful to enforce even for toy projects. they should never be triggered outside of dev, but if they do sometimes it's better to just let it crash.
Making sure the beam is off before crashing would be better though.
Re: The Therac-25 Incident (2021)
#59One of the commenters on the article wrote this: > Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. I was briefly part of an experiment with electronic patient records in an ICU in the early 2000s. My job was to basically babysit the server processing the records in…
Now we have Chipsoft, arguably one of the worst players in the entire IT space that has a near monopoly (around me, anyway) on IT for hospitals. They charge a fortune, produce crap software and the larger they get the less choice there is for the remainder. It is baffling to me that we should be enabling such hostile players.
Re: The Therac-25 Incident (2021)
#60TIL TheDailyWTF is still active. I'd thought it had settled to greatest hits only some years ago.
This story is kind of old.
But also I'm suspicious that this was an AI generated content due to this weird paragraph (one becoming "they"):
It's worth noting that there was one developer who wrote all of this code. They left AECL in 1986, and thankfully for them, no one has ever revealed their identity. And while it may be tempting to lay the blame at their feet—they made every technical choice, they coded every bug—it would be wildly unfair to do that.