Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

221–230 of 307 posts

Re: The Therac-25 Incident (2021)

#221

Post by graypegg »

WalterBright wrote: post #208
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

I think the range of radiation dose might vary too much to make a radiation source a totally isolated system, but trying to keep it as a simple physical lockout, I could imagine part of the start up process involving inserting a small module containing a fuse that breaks at a certain current that could be swapped out for different therapies or something. Could even add a simple spring+electromagnet mechanism that kicks that module out when power gets cut so radiotechs have to at least acknowledge the fuse before start up each time.

I will say that me pretending to know how to best design medical equipment as a web developer is pretty full of myself haha. Highly doubt whatever I'm spouting is a new idea. The idea of working on this sort of high-reliability + high-recoverability systems seems really interesting though!

Re: The Therac-25 Incident (2021)

#222

Post by ChrisMarshallNY »

kulahan wrote: post #207
franktankbank wrote: post #127

Earlier quoted context omitted.

A culture of high-quality engineering, no doubt. Made up of: high quality engineers!

Unfortunately, software developers are the absolute most offensive use of the word "engineer", because 99.9% of the stuff this field makes is a competition to take the most unique approach to a solution, then getting it bandaged together with gum and paperclips. If this industry wants to be respected, it should start trying to be actual engineers. There should be tons and tons of standards which are enforced legally,…

Yeah, this is a fairly classic challenge.

When it comes to safety stuff (like bridge building), there are (and should be) strict licensing requirements. I would have no problem requiring such for work on things like medical equipment, We already require security clearance for things like defense information (unless you're a DOGE bro, I guess). That's a bit different from engineering creds, but it's an example of imposed structure.

But I think that it would be ridiculous to require it for someone that writes a fart app (unless it's a weaponized fart app).

What is in those requirements then becomes a hot potato. There are folks that would insist that any "engineer" be required to know how to use a slide rule, and would ignore modern constructs like LLMs and ML.

I'm not kidding. I know people exactly like that. If they get authority, watch out. They'll only "approve" stuff that they are good at.

On the other hand, if the requirements are LeetCode, then it's useless. A lot of very unqualified people would easily pass, and wreak havoc.

From what I can see, the IEEE seems to have a fairly good grasp on mixing classic structure and current tech. There's some very good people, there, and they are used to working in a structured manner.

But software has developed a YOLO culture. People are used to having almost no structure, and they flit between organizations so rapidly, that it's almost impossible to keep track of who is working on what.

The entire engineering culture needs to be changed. I don't see that being something that will come easily.

I'm big on Structure and Discipline. A lot of it has to do with almost 27 years at a corporation with so much structure that a lot of folks here, would be whimpering under their standing desks.

That structure was required, in order to develop equipment of the Quality they are famous for, but would be total overkill for a lot of stuff.

I do think that we need to impose structure on software supply chains, though. That's not something that will be a popular stance.

Structure is also not cheap. Someone needs to pay for it, and that's when you become a real skunk at the picnic.

Re: The Therac-25 Incident (2021)

#223

Post by darepublic »

sad story. gotta blame canada for this crap. The elements of this story.. hospitals.. a janky attempt at innovation.. passive aggressive denials from otherwise timid demure canadians. Cold grey bureacracy. It all reminds me of the not so great north. The technicians were sipping on their tim hortons slop at the time to make it perfect.

Re: The Therac-25 Incident (2021)

#224

Post by darepublic »

vjvjvjvjghv wrote: post #216
WalterBright wrote: post #208

Earlier quoted context omitted.

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

In general I agree but there is bit more complexity. I work in medical devices and there are plenty of situations where a certain output is ok in some circumstance but deadly in another. That makes a stopgap a little more tricky. I agree with the previous poster that the feedback from the field is lacking a lot. A lot of doctors don’t report problems back because they are used to bad interfaces. And then the feedback…

16000 - 25000 rads right. Not safe under any circumstance?

Re: The Therac-25 Incident (2021)

#225

Post by ponector »

ZaoLahma wrote: post #66
quietbritishjim wrote: post #37

Earlier quoted context omitted.

Right, but if everyone is low quality then there's no one to do that correction. That may seem a bit hypothetical but it can easily happen if you have a company that systematically underpays, which I'm sure many of us don't need to think hard to imagine, in which case they will systematically hire poor developers (because those are the only ones that ever applied).

Replace the "hire poor developers" with "use LLM driven development", and you have the rough outline for a perfect Software Engineering horror movie. It used to be that the poor performers (dangerous hip-shootin' code commitin' cowpokes) were limited in the amount of code that they could produce per time unit, leaving enough time for others to correct course. Now the cowpokes are producing ridiculous amount of code t…

Isn't it a future we are moving to? To hire poor (cheap) developers to do LLM-driven development.

Re: The Therac-25 Incident (2021)

#226

Post by tbossanova »

bigstrat2003 wrote: post #172
edot wrote: post #65

Earlier quoted context omitted.

Isn’t that the pronoun to use when you’re unsure of gender? This article didn’t feel AI-y to me.

The correct neutral pronoun in English is "he", though using "they" has been very popular in recent years due to a mistaken belief that it's sexist to use the male pronoun that way. I wouldn't say it's an AI smell at all.

Not only is "he" not the "correct" pronoun (however you think "correctness" is defined), it also sounds terribly clunky and confusing. "They" is perfectly natural.

Re: The Therac-25 Incident (2021)

#227

Post by ChrisMarshallNY »

kccqzy wrote: post #213
ChrisMarshallNY wrote: post #135

I worked for hardware manufacturers for most of my career, as a software guy. In my experience, hardware people really dis software. It's hard to get them to take it seriously. When something like this happens, they tend to double down on shading software. I have found it very, very difficult to get hardware people to understand that software has a different ruleset and workflow, from hardware. They interpret this as…

Hardware designers benefit from having multiple separate teams to test their product. A chip designer can rely on at least two other teams to test the designed chip, and one of them will be using formal verification. If software also has long release cycles and high cost to remedy mistakes, you bet we would also have multiple testers. In fact that was what happened in the 90s with shrink wrapped software and without…

That was the case for the company that I worked at.

The official QA organization was very powerful, and had no compunctions about stopping an entire product line, for one bug.

When that happened, the department responsible for the bug would find themselves against the wall.

As a result, all the software departments had pretty big teams of testers, who would validate the software, before it was released to the purview of the QA organization.

It could be pretty restricting, but we always felt confident that what we shipped, worked.

Re: The Therac-25 Incident (2021)

#228

Post by onewheeltom »

The manufacturer of the Therac-25, AECL, did not share customer incident reports with other customers when patients were injured. So, the hospitals believed that their incidents were isolated. This may have been legal, but was highly unethical.

Re: The Therac-25 Incident (2021)

#229

Post by DamonHD »

rgoulter wrote: post #63
vemv wrote: post #25

My (tragically) favorite part is, from wikipedia: > A commission attributed the primary cause to generally poor software design and development practices, rather than singling out specific coding errors. Which to me reads as "this entire codebase was so awful that it was bound to fail in some or other way".

Hmm. "poor software design" suggests a high risk that something might go wrong; "poor development practice" suggests that mistakes won't get caught/remedied. By focusing on particular errors, there's the possibility you'll think "problem solved". By focusing on process, you hope to catch mistakes as early as possible.

When trying to make better systems in moderately-critical roles (investment banking, not medicine though) my approach was both try to understand and fix the immediate fault, but also find out if (and fix if so) any systemic issue that would make other related errors likely.

Re: The Therac-25 Incident (2021)

#230

Post by philjohn »

WalterBright wrote: post #208
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

This.

One of the biggest things I see in junior engineers that I mentor (working in backend high throughput, low latency, distributed systems) is not working out all of the various failure modes your system will likely encounter.

Network partitions, primary database outage, caching layer outage, increased latency ... all of these things can throw a spanner in the works, but until you've experienced them (or had a strong mentor guide you) it's all abstract and difficult to see when the happy path is right there.

I've recently entirely re-architected a critical component, and part of this was defense in depth. Stuff is going to go wrong, so having a second or even third line of defense is important.

Post reply on HN