[flagged]
The Therac-25 Incident (2021)
271–280 of 307 posts
Re: The Therac-25 Incident (2021)
#272Earlier quoted context omitted.
Completely safe as long as the block of metal was in place. So you couldn’t just prevent the machine from putting out that much energy, you had to prevent it from doing that without the block in place.
So there should have been an interlocking system
The earlier model that the 25 replaced was all mechanically interlocked. The belief was that software provided that same level of assurance. They performed manual testing but what they weren't able to do was reach a level of speed and fluency with the system to result in the failure modes which caused the issues. Lower hardware costs equals higher profit...
Re: The Therac-25 Incident (2021)
#273> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…
I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…
It's not that I don't think that's important, but I think with failure you always have an issue around needing N+1 checks (please don't take this as an argument against checks though).
The Therac-25 was meant to have a detector of radiation levels to cut things off if a safe value was exceeded, but it didn't work. It could obviously have been improved, but you always have the possibility that "what if our check doesn't work?".
In the case of the Therac-25, if the first initial failures had been reported and investigated, my understanding is (I should make clear I'm not an expert here) it would have made the issues apparent, and it could have been recalled before any of the fatal incidents happened.
In a swiss cheese model of risk, you always want as many layers as possible, so your point about a detector fits in there, but the final layer should always be if an incident does happen, and something gets past all our checks, how can we make it likely that it gets investigated fully by the right person.
Re: The Therac-25 Incident (2021)
#274> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…
I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…
Great point. Earlier in my career, and I think many can see it too, I was very diligent; thorough types, unit tests, defensive programming, assertions at one point, the works.
But this opens up a can of worms, as suddenly you have to deal with every edge case, test for every possible input, etc. This was before fuzz testing, too. Each line of defensive coding, every carefully crafted comment, etc all added to the maintenance burden; I'd even go as far as claim it increased uncertainty, because what if I forgot something?
15 years later and it feels like I'm doing far less advanced stuff (although in hindsight what I did then wasn't all that, but I made it advanced). One issue came up recently; a generic button component would render really tall if no label was given, which happened when a CMS editor did not fill in a label in an attempt to hide it. The knee-jerk response would be to add a check that disallows empty labels, or to not render the button if no label is given, or to use a default button label.
But now I think I'll look at the rendering bug and just... leave the rest. A button with an empty label isn't catastrophic. Writing rules for every possible edge case (empty label, whitespaces, UTF-8 characters escaping the bounds, too long text, too short text, non-text, the list goes on) just adds maintenance and complexity. And it's just a button.
Re: The Therac-25 Incident (2021)
#275Earlier quoted context omitted.
I recently had to argue a junior into leaving the health check frequency alone on an ECS container: the regular log entries annoyed her and she didn't know how to filter logs, so her solution was to take healthchecks down to every five minutes, as just one example of trying to talk to people about the unhappy path.
That sounds more like a disaster waiting to happen than a junior. I find it difficult to believe that she didn't know the purpose of the healthcheck, so it sounds like breaking (someone else's problem) instead of addressing gaps in ability
The junior part there is that this person still believes they can / should read and comprehend all logs themselves. This just isn't viable at scale.
But same with code itself, a junior will have code that is "theirs", a medior/senior will (likely) work at scales where they can't keep it all in their heads. And that's when all the software development best practices come into play.
Re: The Therac-25 Incident (2021)
#276Earlier quoted context omitted.
A culture of high-quality engineering, no doubt. Made up of: high quality engineers!
Unfortunately, software developers are the absolute most offensive use of the word "engineer", because 99.9% of the stuff this field makes is a competition to take the most unique approach to a solution, then getting it bandaged together with gum and paperclips. If this industry wants to be respected, it should start trying to be actual engineers. There should be tons and tons of standards which are enforced legally,…
There are tons and tons of standards, some of which are enforced legally - you can't just supply software for the government, the military, banks, companies etc without certain certifications like ISO 9001, ISO/IEC 27001, etc
Now I'm not an engineer nor at all aware of what these standards actually mean, I'm sure they're pretty common sense and nowhere near as detailed as bridge building standards.
Re: The Therac-25 Incident (2021)
#277Earlier quoted context omitted.
A culture of high-quality engineering, no doubt. Made up of: high quality engineers!
you don't need "high quality engineers" to have high-quality outputs. And vice-versa - lots of places with very high quality engineers produce terribly low-quality software
The article and GP mentions this as well in a roundabout fashion; a high-quality engineer is a waste if the organization around it fails. It's better to have mediocre developers in a mature organization than a hero developer working in the shadows. I've seen a few.
Re: The Therac-25 Incident (2021)
#278Earlier quoted context omitted.
The worst part is that many devlopers think that by not working with high integrity systems, such quality levels don't apply to them. Wrong, any software failure can have huge consequences in someone's life, or company, by preventing some critical flow to take place, corrupting data related to someone's life, professional or medical record, preventing a payment on some specific goods that had to be acquired on that m…
Hey don’t blame developers. It is business who requests features ASAP to cut costs and and then there are customers who don’t want to pay for „ideal software” but rather have every software for free. Most devs and QA workers I know want to deliver best quality software and usually are gold plating stuff anyway.
Business can request it, but it's your job as a software engineer to build quality software; don't shift the blame.
Does a construction engineer blame an architect's wacky designs if a building collapses? No, they either engineer it so it doesn't collapse, convince the architect that it will collapse because physics, or they refuse.
People want to be able to use a bridge for free too, doesn't mean there's no money in it.
As for gold plating, is that really improving software quality, or is that yak shaving / bike shedding?
Re: The Therac-25 Incident (2021)
#279Earlier quoted context omitted.
This is true but there also needs to be good developers as well. It can't just be great process and low quality developer practices. There needs to be: 1/ high quality individual processes (development being one of them), 2/ high quality delivery mechanisms, 3/ feedback loops to improve that quality, 4/ out of band mechanisms to inspect and improve the quality.
I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.
> (this includes almost all of us at some point in time)
I'd say this includes all of us all the time; a good developer never trusts their own work blindly, and spends more time gathering requirements and verifying their and others' work than writing code.
Re: The Therac-25 Incident (2021)
#280Earlier quoted context omitted.
Slow killing software can be made more secure by adding the possibility for human review. Fast killing software is too fast for that.
I'm really trying to understand your point, but I am failing. It sounds like you're saying that you shouldn't care as much about the quality of "slow killing software" because in theory it can be made better in the future? But... it wasn't though? Horizon is a real software system that real developers like you and me built that really killed people. The absolutely terrible quality of it was known about. It was downpl…
I teach the horizon post office scandal in my database courses. And my takeaway is, that software fails. And if people's lives are involved, an audit trail is paramount.
In slowly killing software the audit trail might be faster than the killing. In fast killing software, the audit trail isn't.