Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

191–200 of 307 posts

Re: The Therac-25 Incident (2021)

#191

Post by f1shy »

ChrisMarshallNY wrote: post #120
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I worked for a company that manufactured some of the highest-Quality photographic and scientific equipment that you can buy. It was expensive as hell, but our customers seemed to think it was worth it. > It's the end result of a process In my experience, it's even more than that. It's a culture .

It is a culture. No doubt. And certainly not only processes. I work in a company where there are literally processes for everything. For every engineer doing actual work (requirement engineering, architecture, coding, testing) there are at least 3!doing processes. The SW we make is the ultimative piece of shit, late, expensive, and to the brim full of bugs. So process is importannt, but good engineering, and culture also.

Re: The Therac-25 Incident (2021)

#192

Post by therein »

bongodongobob wrote: post #188
skinwill wrote: post #64

Earlier quoted context omitted.

Around here we have Epic. If you want a good scare, look up their corporate Willy Wonka-esq jail/campus and their policy of zero remote work.

Their hiring is insane too. Years ago I applied for a Sys admin I position. After 3 rounds of interviews, they gave me a test to take. Except it was something you'd give a senior dev. They specced out a fake language, super esoteric too, it was like a combination of brainfuck and assembly, and asked me to solve problems with it, and debug code written in it. I could have toughed my way through it, but I was so dissat…

I remember my friend was looking at applying Epic. They asked him to send his SAT scores. We had a good chuckle about it.

He had finished undergrad 5+ years prior and had continuous industry experience.

Re: The Therac-25 Incident (2021)

#193

Post by snerbles »

bell-cot wrote: post #101
vorgol wrote: post #39

Earlier quoted context omitted.

I was going to recommend that exact podcast episode but you beat me to it. Totally worth listening, especially if you're interested in software bugs. Another interesting fact mentioned in the podcast is that the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. Excellent demonstration of the Swiss Cheese Model: https:…

>> the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed. > the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. #1 virtue of electromechanical failsafes is that their conception, design, implementation, and failure mode…

When I worked at an industrial integrator, we had a hard requirement for hard-wired e-stop circuits run by safety relays separate from the PLC. Sometimes we had to deal with dangerous OEM equipment that had software interlocks, and the solution was usually just to power the entire offending device down when someone hit an e-stop or opened a guarding panel.

About a decade ago a rep from Videojet straight up lied to us about their 30W CO2 marking laser having a hardware interlock. We found out when - in true Therac-25 fashion - the laser kept triggering despite the external e-stop being active due to a bug in their HMI touch panel. No one noticed until it eventually burned through the lens cap. In reality the interlock was a separate kit, and they left it out to reduce the cost for their bid to the customer. That whole incident really soured my opinion of them and reminded me of just how bad software "safety" can get.

Re: The Therac-25 Incident (2021)

#194

Post by mrguyorama »

bobmcnamara wrote: post #114
mnw21cam wrote: post #86

Earlier quoted context omitted.

There are plenty of "car allegedly accelerated by itself" incidents, and usually the root cause is the driver mistakenly pressing the accelerator pedal when they think they're pressing the brake pedal. And then swearing blind afterwards that they were braking as hard as they possibly could but the car kept surging forwards.

Time and time again the introduction of electronic throttle control has spiked the number of reported unintended acceleration incidents. There's a chart here that shows it clearly for Toyota's rollout: https://www.embedded.com/unintended-acceleration-and-other-e...

That chart is not clear.

The Camry, the Solara, and the RAV4 are all the exact same engine hardware and software, at nearly all times. Especially the Solara, which is just a Camry with a shinier body. It uses an identical engine, throttle body, and ECU flash and is even considered a "Camry Solara".

That Camry "Unintended accelerations" jumped while Solara didn't means that it isn't the hardware. Instead, they all started at the same time, about 2002. Similarly, the Scion TC is also literally the exact same hardware, software, and throttle body as a Camry. The entire Scion line is just Toyota software and hardware in a different body shell.

Indeed, if you look at the Corolla, the jump in unintended acceleration cases start with mechanical throttle bodies still the norm, and do not change with the switch to electronic throttle control.

IMO this graphic handily shows how the media affects average people. The media went on a large blitz about how Toyota was unsafe now that they had electronic throttle bodies, and so owners of those cars complained, but the average consumer does not realize that the Solara, Camry, and Scion TC are all identical vehicles under the body shell and don't realize that they "should" also be complaining about those cars if the problem was actually caused by the electronic throttle body code or design.

Important note: People who report unintended acceleration events like this almost always say that the brakes didn't stop it. That seems.... hard to believe. The brakes on all Toyotas are fully hydraulic and cannot fail electronically. All toyota vehicles in that chart (maybe excluding some Tacomas and the top line Lexus model) have brake capability far exceeding their engine power. A V6 Camry can be at wide open throttle but hard application of the brakes would still overpower that engine with no problem.

Interestingly the NHTSA's opinion is that the Toyota models in the graphic ARE defective: They allow faulty or improperly installed weather mats to interfere with the pedals. Toyota also believed this take, as they kept their handling of fixing this defect off the books, and changed the pedal positioning in their newer models. They eventually fined Toyota over a billion dollars for their handling of this situation, and claimed there was another "sticky pedals" problem that they were covering up.

Re: The Therac-25 Incident (2021)

#196

Post by mrguyorama »

rvz wrote: post #8

We're more likely to get a similar incident like this very quickly if we continue with the cult of 'vibe-coding' and throwing away basic software engineering principles out of the window as I said before. [0] Take this post-mortem here [1] as a great warning and which also highlights exactly what could go horribly wrong if the LLM misreads comments. What's even more scarier is each time I stumble across a freshly min…

God that "post mortem" is such a portent of things to come. I've seen this exact problem path happen locally nearly any time I use claude. It very obviously just picks what it should put where based on weighted random chance, and that random chance is going to not go in your favor at some point, in a way that no amount of training or job experience can help with, because no, a human would not have made this mistake.

This is the kind of mistake that fails people out of CS101; It's obvious that the student is just manipulating symbols they don't really "get" rather than modifying code. Throwing the chinese room thought experiment at your code base is bad engineering.

Re: The Therac-25 Incident (2021)

#197

Post by w10-1 »

This is not the example readers need to understand, because the failures were so rudimentary and systemic that it seems "good process" is the answer.

Having written and validated both FDA and CLIA software, I'd suggest that process is never sufficient.

Plenty of well-meaning people will create and follow incomplete plans and hand-wave away issues when they sign off -- particularly people who gravitate towards rule-based, formulaic work in a hierarchy.

You need people both capable of and willing to seriously question whether proof is really proof, and who will stand up for some random patient in the distant future over their boss and colleagues on a deadline -- and yet they cannot be oppositional or egotistical, and must have deep insight into the subject matter.

It's really, really hard to find those people.

Re: The Therac-25 Incident (2021)

#198

Post by jacquesm »

bongodongobob wrote: post #188
skinwill wrote: post #64

Earlier quoted context omitted.

Around here we have Epic. If you want a good scare, look up their corporate Willy Wonka-esq jail/campus and their policy of zero remote work.

Their hiring is insane too. Years ago I applied for a Sys admin I position. After 3 rounds of interviews, they gave me a test to take. Except it was something you'd give a senior dev. They specced out a fake language, super esoteric too, it was like a combination of brainfuck and assembly, and asked me to solve problems with it, and debug code written in it. I could have toughed my way through it, but I was so dissat…

That sounds like a filter selecting for people without any self esteem but mad programming chops. Good you passed.

Re: The Therac-25 Incident (2021)

#199

Post by NortySpock »

greazy wrote: post #68
elric wrote: post #43

One of the commenters on the article wrote this: > Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. I was briefly part of an experiment with electronic patient records in an ICU in the early 2000s. My job was to basically babysit the server processing the records in…

It's still an issue. I've heard stories of EMR system going down forcing staff to use pen and paper. It boggles my mind that such systems don't have redundancy. These are commercial products being deployed.

The redundancy is pen and paper. The EMR just helps teams coordinate faster, pull up records faster, etc.

When I worked at Cerner years ago (now owned by Oracle), there were rumors that the Cerner EMR still could barely handle DST* spring forward, but could not handle DST fall back (where the 01:00 hour is repeated) -- you had do preemptively switch to pen-and-paper for the hours around the switch. I assume this was because someone back in the initial database design used local time instead of UTC for some critical patient-care timestamp fields in the database, and then had a bear of a time getting reliable times out of the database during the witching hour.

* Daylight Saving Time in the USA. And yes, everyone in the USA changes non-networked clocks twice a year because of some "brilliant idea" someone shoved through Congress in 1974.

EDIT: I wonder if Cerner finally fixed it?

Re: The Therac-25 Incident (2021)

#200

Post by csours »

To me, the Therac incident is the poster child for a category I call 'context change error'.

Some of the controls were 'born' in a world of hardware interlocks, and so the engineers used the frame of mind where hardware interlocks exist.

Some time later, the interlocks were replaced with software controls. Since everything had worked before, all the software had to do was what worked before.

But it is VERY difficult to challenge all of your assumptions about what "working" means.

---

This is also a good reminder that work is done by people and teams, not corporations. That is - just because somebody knows the fine details, that does not mean that the corporation knows the fine details.

Post reply on HN