Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

211–220 of 307 posts

Re: The Therac-25 Incident (2021)

#211

Post by I_dream_of_Gen1 »

snerbles wrote: post #193
bell-cot wrote: post #101

Earlier quoted context omitted.

>> the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed. > the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. #1 virtue of electromechanical failsafes is that their conception, design, implementation, and failure mode…

When I worked at an industrial integrator, we had a hard requirement for hard-wired e-stop circuits run by safety relays separate from the PLC. Sometimes we had to deal with dangerous OEM equipment that had software interlocks, and the solution was usually just to power the entire offending device down when someone hit an e-stop or opened a guarding panel. About a decade ago a rep from Videojet straight up lied to us…

[dead]

Re: The Therac-25 Incident (2021)

#212

Post by I_dream_of_Gen1 »

kulahan wrote: post #207
franktankbank wrote: post #127

Earlier quoted context omitted.

A culture of high-quality engineering, no doubt. Made up of: high quality engineers!

Unfortunately, software developers are the absolute most offensive use of the word "engineer", because 99.9% of the stuff this field makes is a competition to take the most unique approach to a solution, then getting it bandaged together with gum and paperclips. If this industry wants to be respected, it should start trying to be actual engineers. There should be tons and tons of standards which are enforced legally,…

[dead]

Re: The Therac-25 Incident (2021)

#213

Post by kccqzy »

ChrisMarshallNY wrote: post #135

I worked for hardware manufacturers for most of my career, as a software guy. In my experience, hardware people really dis software. It's hard to get them to take it seriously. When something like this happens, they tend to double down on shading software. I have found it very, very difficult to get hardware people to understand that software has a different ruleset and workflow, from hardware. They interpret this as…

Hardware designers benefit from having multiple separate teams to test their product. A chip designer can rely on at least two other teams to test the designed chip, and one of them will be using formal verification. If software also has long release cycles and high cost to remedy mistakes, you bet we would also have multiple testers. In fact that was what happened in the 90s with shrink wrapped software and without easy updates.

Re: The Therac-25 Incident (2021)

#214

Post by jopsen »

0xDEAFBEAD wrote: post #98
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".

I'd agree, it's super easy to think such errors won't happen had they just used a fairly safe language and sane architecture. Or unit test, race detectors, etc.

I suspect that few organizations that do all that, have a process/culture of ignoring bugs in the wild -- and those that do have such complicated domains that explaining the error is hard.

Software best practices today would probably also involve sending metrics, logs, error reports, etc.

That said, it's still extremely easy get embrace a culture were unexplainable errors are ignored. Especially in a cloud environment.

Re: The Therac-25 Incident (2021)

#215

Post by fogzen »

Almost. It’s a process problem. But the process is a step above the organization. It’s a socio-economic process that incentivizes these problems. It’s capitalism that’s the process problem. That’s the process that introduces the problem into the organization. Without the government regulators making them test nothing would have even been done at all. Because the problem is the organization exists within a framework that pits it against safety. Safety is at odds with what the organization is tasked to do within the process that it exists in.

Re: The Therac-25 Incident (2021)

#216

Post by vjvjvjvjghv »

WalterBright wrote: post #208
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

In general I agree but there is bit more complexity. I work in medical devices and there are plenty of situations where a certain output is ok in some circumstance but deadly in another. That makes a stopgap a little more tricky.

I agree with the previous poster that the feedback from the field is lacking a lot. A lot of doctors don’t report problems back because they are used to bad interfaces. And then the feedback gets filtered through several layers of sales reps and product management. So a lot of info gets lost and fixes that could be simple won’t get done.

In general when you work in medical you are so overwhelmed by documentation and regulation that there isn’t much time left to do proper engineering. The FDA mostly looks at documentation done right and less at product done right.

Re: The Therac-25 Incident (2021)

#217

Post by vjvjvjvjghv »

Fr3dd1 wrote: post #34
AdamN wrote: post #16

Earlier quoted context omitted.

This is true but there also needs to be good developers as well. It can't just be great process and low quality developer practices. There needs to be: 1/ high quality individual processes (development being one of them), 2/ high quality delivery mechanisms, 3/ feedback loops to improve that quality, 4/ out of band mechanisms to inspect and improve the quality.

I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.

“ This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process”

That’s a horrible take. There is no amount of reviews, guidelines and documentation that can compensate for low quality devs. You can’t throw garbage into the pipeline and then somehow process it to gold.

Re: The Therac-25 Incident (2021)

#218

Post by I_dream_of_Geni »

roeles wrote: post #125
0xDEAFBEAD wrote: post #98

Earlier quoted context omitted.

Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".

One instance that crosses my mind often is the airbus a320 incident at Hamburg in 2008. Everything was done right there, but the requirements were wrong. Despite all the procedures and tests, the software still managed to endanger the lives of the passengers.

Speaking of Airbus, They 'lost' 3-4 different aircraft (from 1988 to 2015) which crashed during development, or, spectacularly during their first airshow. Never slowed down their customers at ALL, and to this day, Boeing has never lost one new commercial airliner in those same circumstances. Yet, Boeing gets all the hate. smh

Re: The Therac-25 Incident (2021)

#219

Post by ponector »

ChrisMarshallNY wrote: post #165
ChrisMarshallNY wrote: post #138

Earlier quoted context omitted.

Not really. Good developers are a necessary ingredient of a much larger recipe. People think that a good process means you can toss in crap developers, or that great developers mean that you can have a bad process. In my experience, I worked for a 100-year-old Japanese engineering company that had a decades-long culture of Quality. People stayed at that company for their entire career, and most of them were top-shelf…

It really is funny how any discussion of improving software quality becomes unpopular, here.

Not only here. Everyone wants to use a quality products, but no almost one is committed to deliver quality products/services.

No one is working on quality, everyone works on new features. There is usually no incentive to increase quality, to improve speed, performance, etc.

Re: The Therac-25 Incident (2021)

#220

Post by WalterBright »

vjvjvjvjghv wrote: post #216
WalterBright wrote: post #208

Earlier quoted context omitted.

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

In general I agree but there is bit more complexity. I work in medical devices and there are plenty of situations where a certain output is ok in some circumstance but deadly in another. That makes a stopgap a little more tricky. I agree with the previous poster that the feedback from the field is lacking a lot. A lot of doctors don’t report problems back because they are used to bad interfaces. And then the feedback…

At Boeing there's a required "failure analysis" document listing all the failure modes and why they won't cause a crash by themselves.
Post reply on HN