Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

281–290 of 307 posts

Re: The Therac-25 Incident (2021)

#281

Post by Cthulhu_ »

ZaoLahma wrote: post #66
quietbritishjim wrote: post #37

Earlier quoted context omitted.

Right, but if everyone is low quality then there's no one to do that correction. That may seem a bit hypothetical but it can easily happen if you have a company that systematically underpays, which I'm sure many of us don't need to think hard to imagine, in which case they will systematically hire poor developers (because those are the only ones that ever applied).

Replace the "hire poor developers" with "use LLM driven development", and you have the rough outline for a perfect Software Engineering horror movie. It used to be that the poor performers (dangerous hip-shootin' code commitin' cowpokes) were limited in the amount of code that they could produce per time unit, leaving enough time for others to correct course. Now the cowpokes are producing ridiculous amount of code t…

This is why at every software project I've done in the past 15 odd years, steps were taken to prevent this in an automated and standardized fashion; code reviews of course, but they're more for functionality. Unit test requirements, integration / end-to-end tests based on acceptance criteria, visual regression tests, linting, type systems, OTAP, CI/CD, audit log via Git and standardized commit messages, etc etc etc.

My job hasn't significantly changed with AI, as AI generated code still has to pass all the hurdles I've set up while setting up this project.

Re: The Therac-25 Incident (2021)

#282

Post by benrutter »

msy wrote: post #246
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I couldn't disagree more. Outside of exotic scenarios where things like formal proofing are possible and economically viable I've never seen a process that prevents bugs, only culture. Good engineering cultures are also often ones with well defined and tested processes and good testing practices but it's the culture and people giving a shit that makes the difference, not the other way around. Good product cultures ar…

> It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing.

I think the bit I quoted, especially if you read in the context of the article, is talking about culture. I.e. it's talking about a process that informs software development, management and sales. Things like formal proofing and type systems are the exact kind of processes that aren't what it's talking about.

I kind of agree with you though about the process/culture distinction - ultimately, if you don't have a culture where people actively care about improving reliability, any process is just gonna become a tick-box exercise to appease management.

Re: The Therac-25 Incident (2021)

#283

Post by benrutter »

rowanG077 wrote: post #240
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I think the opposite. The only reason software quality emerges is because of good developers. It's a prerequisites. Process helps good developers deliver quality. But there is no process that allows a team of bad developers deliver quality. you can't squeeze blood from a stone.

That's true, but even so - great developers still make mistakes, and if they don't hear about production errors because of a breakdown in the customer communications from sales etc, then those mistakes will never be fixed.

It's not that great developers aren't necessary for software quality, more that they aren't sufficient.

Re: The Therac-25 Incident (2021)

#285

Post by elric »

superjan wrote: post #176
elric wrote: post #43

One of the commenters on the article wrote this: > Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. I was briefly part of an experiment with electronic patient records in an ICU in the early 2000s. My job was to basically babysit the server processing the records in…

It”s worthwhile to mention that in the US and EU EMRs are generally not considered Medical Devices and are therefore not subject to a lot of regulations. https://www.medicaleconomics.com/view/what-if-emrs-were-clas...

That is disturbing, but it does explain a lot.

Re: The Therac-25 Incident (2021)

#286

Post by donatj »

elric wrote: post #285
superjan wrote: post #176

Earlier quoted context omitted.

It”s worthwhile to mention that in the US and EU EMRs are generally not considered Medical Devices and are therefore not subject to a lot of regulations. https://www.medicaleconomics.com/view/what-if-emrs-were-clas...

That is disturbing, but it does explain a lot.

Leaves the space decently open to interruption at least, and from my understanding it kind of needs it.

Re: The Therac-25 Incident (2021)

#287

Post by SCdF »

maweki wrote: post #280
SCdF wrote: post #267

Earlier quoted context omitted.

I'm really trying to understand your point, but I am failing. It sounds like you're saying that you shouldn't care as much about the quality of "slow killing software" because in theory it can be made better in the future? But... it wasn't though? Horizon is a real software system that real developers like you and me built that really killed people. The absolutely terrible quality of it was known about. It was downpl…

I teach the horizon post office scandal in my database courses. And my takeaway is, that software fails. And if people's lives are involved, an audit trail is paramount. In slowly killing software the audit trail might be faster than the killing. In fast killing software, the audit trail isn't.

Yes, the audit trail that should exist is part of the package. Or more generically, Horizon should have had enough instrumentation, combined with adequate robustness, where they could detect the issues the lack of robustness caused, and resolve those issues without people dying.

My core point is that if you're designing a system, *any system*, you should be thinking about what is required to produce safe software. It isn't just "well I don't work on medical devices that shoot radiation at people, so I don't need to worry"[1]. You still need to worry, you just solve those problems in different ways. It's not just deaths either, it's PII leakage, it's stalking and harassment enablement, it's privilege escalation, etc.

[1] I have heard this, or a variation of this, from dozens of people over the my career. This is my core bug bear about Therac-25, is that it allows people to think this way, and divest themselves of responsibility. I am very happy to hear you are teaching a course about Horizon, because it's a much more grounded example that devs will hopefully see themselves in more. If your course is publicly available btw, I'd love to read it.

Re: The Therac-25 Incident (2021)

#288

Post by donatj »

V__ wrote: post #94
hahn-kev wrote: post #49

Earlier quoted context omitted.

From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.

> Engineers are legally obligated to report unsafe conduct, activities or behaviours of others that could pose a risk to the public or the environment. [1] If software "engineers" want to be taken seriously, then they should also have the obligation to report unsafe/broken software and refuse to ship unsafe/broken software. The developers are just as much to blame as the post office: > Fujitsu was aware that Horizon…

I have worked in this industry for 20 years and never met a piece of software I would deem "safe". It's all duct tape and spit. All of it.

I have had software professionally audited by third parties more than a few times, and they basically only ever catch surface level bugs. Recently, the same we the audit finished we independently found a pretty obvious sql injection flaw.

I think the danger is not in producing unsafe software. The real danger is in thinking it can ever can be safe. It cannot be, and anyone who tells you otherwise is a snake oil salesman.

If your life depends on software, you are one bit flip from death.

Re: The Therac-25 Incident (2021)

#289

Post by WalterBright »

bombcar wrote: post #250
WalterBright wrote: post #209

Earlier quoted context omitted.

> And PHB's all know that picking up nickles in front of the steamroller is how you get to the C-suite. Blaming it on PHB's is a mistake. There were no engineering classes in my degree program about failsafe design. I've known too many engineers who were insulted by my insinuations that their design had unacceptable failure modes. They thought they could write software that couldn't possibly fail. They'd also tell me…

Failsafe design is actually really fun when you start looking at all the scenarios and such. But one key component is that IF a failsafe is triggered, it needs to be investigated as if it killed someone; because it should NEVER have triggered. Without that part of the cycle, eventually the failsafe is removed or bypassed or otherwise ineffective, and the next incident will get you.

Most airplane crashes are due to multiple failures. The accidents are investigated, and each failure is addressed and fixed.

The result is incredible safety.

Re: The Therac-25 Incident (2021)

#290

Post by maweki »

SCdF wrote: post #287
maweki wrote: post #280

Earlier quoted context omitted.

I teach the horizon post office scandal in my database courses. And my takeaway is, that software fails. And if people's lives are involved, an audit trail is paramount. In slowly killing software the audit trail might be faster than the killing. In fast killing software, the audit trail isn't.

Yes, the audit trail that should exist is part of the package. Or more generically, Horizon should have had enough instrumentation, combined with adequate robustness, where they could detect the issues the lack of robustness caused, and resolve those issues without people dying. My core point is that if you're designing a system, *any system*, you should be thinking about what is required to produce safe software. It…

It's just a course about database design and in the first seminar we look at different news stories that have something to do with databases, like trump putting some random Italian chef on an international sanction list should make us think about primary keys and identifying people.

And the horizon post office scandal is the last and most poignant example that real people are affected by the systems we build and the design decisions we make. That sometimes easy to forget.

Post reply on HN