Earlier quoted context omitted.
The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.
But there is still a difference here. Provenance and proper traceability would have allowed the subpostmasters to show their innocence and prove the system failable. In the Therac-25 case, the killing was quite immediate and it would have happened even if the correct radiation dose was recorded.
The Therac-25 Incident (2021)
111–120 of 307 posts
Re: The Therac-25 Incident (2021)
#112> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…
Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".
Re: The Therac-25 Incident (2021)
#113It's interesting to compare this with the Post Office Scandal in the UK. Very different incidents, but reading this, there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong". For developers, this is a hilariously silly thing, but for non-developers looking at it from the outside, they don't have the capability or training to understand that software can be this f…
> there is arguably a root assumption in both cases that people made, which is that "the software can't be wrong" I think in this case, the thought process was based on the experience with older, electro-mechanical machines where the most common failure modern was parts wearing out. Since software can, indeed, not "wear out", someone made the assumption that it was therefore inherently more reliable.
Bureaucracy being (per Graeber 2006) something like the ritual where by means of a set of pre-fashioned artifacts for each other's sake we all operate at 2% of our normal mental capacities and that's how modern data-driven, conflict-averse societies organize work and distribute resources without anyone being able to have any complaints listened to.
>Bureaucracies public and private appear—for whatever historical reasons—to be organized in such a way as to guarantee that a significant proportion of actors will not be able to perform their tasks as expected. It also exemplifies what I have come to think of the defining feature of a utopian form of practice, in that, on discovering this, those maintaining the system conclude that the problem is not with the system itself but with the inadequacy of the human beings involved.
Most places where a computer system is involved in the administration of a public service or something of the caliber, has that been a grassroots effort, hey computers are cool and awesome let's see what they change? No, it's something that's been imposed in the definitive top-down manner of XX century bureaucracies. Remember the cohort of people who used to become stupid the moment a "thinking machine" was powered within line of sight (before the last uncomputed generation retired and got their excuse to act dumb for the rest of it)? Consider them in view the literally incomprehensible number of layers that any "serious" piece of software consists of; layers which we're stuck producing more of, when any software professional knows the best kind of software is less of it.
But at least it saves time and the forest, right? Ironically, getting things done in a bureaucratic context with less overhead than filling out paper forms or speaking to human beings, makes them even easier to fuck up. And then there's the useful fiction of "the software did it" that e.g. "AI agents" thing is trying to productize. How about they just give people a liability slider in the spinup form, eh, but nah.
Wanna see a miracle? A miracle is when people hype each other into pretending something impossible happened. To the extent user-operated software is involved in most big-time human activities, the daily miracle is how it seems to work well enough, for people to be able to pretend it works any good at all. Many more than 3 such cases. But of course remembering the catastrophal mistakes of the past can be turned into a quaint fun-time activity. Building things that empower people to make less mistakes, meanwhile, is a little different from building artifacts for non-stop "2% time".
Re: The Therac-25 Incident (2021)
#114Earlier quoted context omitted.
There was a news story from Norway last year where a car allegedly accelerated by itself, causing the car to fall off the second floor of a parking garage and kill the driver.
There are plenty of "car allegedly accelerated by itself" incidents, and usually the root cause is the driver mistakenly pressing the accelerator pedal when they think they're pressing the brake pedal. And then swearing blind afterwards that they were braking as hard as they possibly could but the car kept surging forwards.
There's a chart here that shows it clearly for Toyota's rollout:
https://www.embedded.com/unintended-acceleration-and-other-e...
Re: The Therac-25 Incident (2021)
#115The full 1993 report linked in the article has an intetesting statement regarding software developer certfication in the "Lessons learned" chapter: > Taking a couple of programming courses or programming a home computer does not qualify anyone to produce safety-critical software. Although certification of software engineers is not yet required, more events like those associated with the Therac-25 will make such certi…
Re: The Therac-25 Incident (2021)
#116I'd be interested in knowing how many of y'all are being taught about this sort of thing in college ethics/safety/reliability classes. I was taught about this in engineering school, as part of a general engineering course also covering things like bathtub reliability curves and how to calculate the number of redundant cooling pumps a nuclear power plant needs. But it's a long time since I was in college. Is this sort…
It was taught in a first year software ethics class on my Computer Science programme. Back in 2010. I'm wondering if they still do
Re: The Therac-25 Incident (2021)
#117Re: The Therac-25 Incident (2021)
#118One of the commenters on the article wrote this: > Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. I was briefly part of an experiment with electronic patient records in an ICU in the early 2000s. My job was to basically babysit the server processing the records in…
It's still an issue. I've heard stories of EMR system going down forcing staff to use pen and paper. It boggles my mind that such systems don't have redundancy. These are commercial products being deployed.
The other theory is there are soo many bureaucratic hoops to jump through in order to make anything in the medical space, that no one does it willingly.
Re: The Therac-25 Incident (2021)
#119They can't. There was a single developer, he left, no tests existed, no one understood the mess to confidently make changes. At this point you can either lie your way through the regulators or scrap the product altogether.
I've seen this kind of devs and companies running their software in regulated industries like in the therac incident, just now we are in the year 2025. I left because i understood that it's a criminal charge waiting to happen.
Re: The Therac-25 Incident (2021)
#120> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…
> It's the end result of a process
In my experience, it's even more than that. It's a culture.