Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

91–100 of 307 posts

Re: The Therac-25 Incident (2021)

#91

Post by throwaway0261 »

One of the comments said this:

> That standard [IEC 62304] is surrounded by other technical reports and guidances recognized by the FDA, on software risk management, safety cases, software validation. And I can tell you that the FDA is very picky, when they review your software design and testing documentation. For the first version and for every design change.

> That’s good news for all of us. An adverse event like the Therac 25 is very unlikely today.

This is a case where regulation is a good thing. Unfortunately I see a trend lately where almost any regulation is seen as something stopping innovation and business growth. There are room for improvements and some areas are over regulated, but we don't want a "DOGE" chainsaw to regulations without knowing what the consequences are.

Re: The Therac-25 Incident (2021)

#92

Post by rcxdude »

Fr3dd1 wrote: post #34
AdamN wrote: post #16

Earlier quoted context omitted.

This is true but there also needs to be good developers as well. It can't just be great process and low quality developer practices. There needs to be: 1/ high quality individual processes (development being one of them), 2/ high quality delivery mechanisms, 3/ feedback loops to improve that quality, 4/ out of band mechanisms to inspect and improve the quality.

I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.

This only works with enough good developers involved in the process. I've seen how the sausage is made, and code quality is often shockingly low in these applications, just in ways that don't set off the metrics (or they do, but they can bend the process to wave them away). Also, the process often makes it very hard to fix latent problems in the software, so it rarely gets better over time, either.

Re: The Therac-25 Incident (2021)

#93

Post by softwaredoug »

Safety problems are almost never about one evil / dumb person and frequently involve confusing lines of responsibility.

Which makes me very nervous about AI generated code and people who don’t clam human authorship. The bug that creeps in where we scapegoat the AI isn’t gonna cut it in a safety situation.

Re: The Therac-25 Incident (2021)

#94

Post by V__ »

hahn-kev wrote: post #49
SCdF wrote: post #41

Earlier quoted context omitted.

The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.

> Engineers are legally obligated to report unsafe conduct, activities or behaviours of others that could pose a risk to the public or the environment. [1]

If software "engineers" want to be taken seriously, then they should also have the obligation to report unsafe/broken software and refuse to ship unsafe/broken software. The developers are just as much to blame as the post office:

> Fujitsu was aware that Horizon contained software bugs as early as 1999 [2]

[1] https://engineerscanada.ca/news-and-events/news/the-duty-to-...

[2] https://en.wikipedia.org/wiki/British_Post_Office_scandal

Re: The Therac-25 Incident (2021)

#95

Post by aaron695 »

benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

[dead]

Re: The Therac-25 Incident (2021)

#96

Post by speed_spread »

sonicggg wrote: post #88
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

Not sure why the article is focusing so much on software development. That was just a piece of the problem. The entire product had design flaws. When the FDA for involved, the company wasn't just told to make software updates.

Yet It doesn't take much to swamp a team of good developers. A poorly defined project, mismatched requirements, sent to production too early and then put in support mode with no time planned to plug the holes... There's only so much smart technicians can do when the organization is broken.

Re: The Therac-25 Incident (2021)

#98

Post by 0xDEAFBEAD »

benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".

Re: The Therac-25 Incident (2021)

#99

Post by tbossanova »

greatgib wrote: post #60
mellosouls wrote: post #24

TIL TheDailyWTF is still active. I'd thought it had settled to greatest hits only some years ago.

This story is kind of old. But also I'm suspicious that this was an AI generated content due to this weird paragraph (one becoming "they"): It's worth noting that there was one developer who wrote all of this code. They left AECL in 1986, and thankfully for them, no one has ever revealed their identity. And while it may be tempting to lay the blame at their feet—they made every technical choice, they coded every bug—…

That is 100% standard english, dude. I feel like I might have read that exact sentence 20 years ago...

Re: The Therac-25 Incident (2021)

#100

Post by pjmlp »

quietbritishjim wrote: post #37
Fr3dd1 wrote: post #34

Earlier quoted context omitted.

I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.

Right, but if everyone is low quality then there's no one to do that correction. That may seem a bit hypothetical but it can easily happen if you have a company that systematically underpays, which I'm sure many of us don't need to think hard to imagine, in which case they will systematically hire poor developers (because those are the only ones that ever applied).

The correction is done by the "lucky" souls doing the onsite, customer facing roles, for the offshoring delivery. Experience from a friend....
Post reply on HN