Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

251–260 of 307 posts

Re: The Therac-25 Incident (2021)

#251

Post by ramses0 »

FuriouslyAdrift wrote: post #142
michaelt wrote: post #6

I'd be interested in knowing how many of y'all are being taught about this sort of thing in college ethics/safety/reliability classes. I was taught about this in engineering school, as part of a general engineering course also covering things like bathtub reliability curves and how to calculate the number of redundant cooling pumps a nuclear power plant needs. But it's a long time since I was in college. Is this sort…

A big thing that was emphasized in my computer engineering courses at Purdue in the early 90s with regards to machine interfaces was hysteresis. A machine has a RANGE of behaviors throughout it's operating area that might not be accounted fro in your programing and you must take that into consideration (i.e. a robotic arm or electric motor doesn't just 'stop' instantly). Analog systems do not behave like computers.

The "IBM Black Team Debugs a Tape Drive" story comes to mind: https://www.penzba.co.uk/GreybeardStories/TheBlackTeam.html

Re: The Therac-25 Incident (2021)

#252

Post by _kb »

dezgeg wrote: post #206
zackmorris wrote: post #184

Our power went off a couple off weeks ago due to wind probably knocking a branch into a power line. Now our Frigidaire microwave runs with the door open. Supposedly there are mechanical switches that prevent that, but evidently "modern" microwaves can control the gun through the logic board. The engineering failures that led to this, from conceptual to design to internal control, boggle my mind. I'm not even sure whe…

It's possible that it's just operating the lights and turntable without actually cooking - if you search HN you find that failure mode.

Stand next to it with a chocolate bar in your pocket. I believe that’s the canonical way to check if it’s active.

Re: The Therac-25 Incident (2021)

#253

Post by drfolger »

WalterBright wrote: post #208
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

[dead]

Re: The Therac-25 Incident (2021)

#254

Post by simoncion »

bongodongobob wrote: post #189
zackmorris wrote: post #184

Our power went off a couple off weeks ago due to wind probably knocking a branch into a power line. Now our Frigidaire microwave runs with the door open. Supposedly there are mechanical switches that prevent that, but evidently "modern" microwaves can control the gun through the logic board. The engineering failures that led to this, from conceptual to design to internal control, boggle my mind. I'm not even sure whe…

I have a microwave from the early 80s. If I stick a pencil in the door latch I can get it to run with the door open as well. It's not the demon core. Just don't stick your head in it when it's running.

> If I stick a pencil in the door latch I can get it to run with the door open as well.

"The safety interlocks don't work when the operator intentionally goes out of his way to defeat them." isn't a concern. There's only so much you can do to prevent someone who's dedicated to disabling them.

"The safety interlocks fail dangerous because of an unexpected power cut." is a huge concern. What else did the manufacturer skimp on, or -worse- simply fail to understand was important to do for the safety of the operator of the device?

Re: The Therac-25 Incident (2021)

#255

Post by fsulkner »

WalterBright wrote: post #220
vjvjvjvjghv wrote: post #216

Earlier quoted context omitted.

In general I agree but there is bit more complexity. I work in medical devices and there are plenty of situations where a certain output is ok in some circumstance but deadly in another. That makes a stopgap a little more tricky. I agree with the previous poster that the feedback from the field is lacking a lot. A lot of doctors don’t report problems back because they are used to bad interfaces. And then the feedback…

At Boeing there's a required "failure analysis" document listing all the failure modes and why they won't cause a crash by themselves.

[dead]

Re: The Therac-25 Incident (2021)

#256

Post by technofiend »

philjohn wrote: post #230
WalterBright wrote: post #208

Earlier quoted context omitted.

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

This. One of the biggest things I see in junior engineers that I mentor (working in backend high throughput, low latency, distributed systems) is not working out all of the various failure modes your system will likely encounter. Network partitions, primary database outage, caching layer outage, increased latency ... all of these things can throw a spanner in the works, but until you've experienced them (or had a str…

I recently had to argue a junior into leaving the health check frequency alone on an ECS container: the regular log entries annoyed her and she didn't know how to filter logs, so her solution was to take healthchecks down to every five minutes, as just one example of trying to talk to people about the unhappy path.

Re: The Therac-25 Incident (2021)

#257

Post by ChrisMarshallNY »

herval wrote: post #244
ChrisMarshallNY wrote: post #167

Earlier quoted context omitted.

I guess we see things differently. They don't need to be especially talented engineers, but, in my experience (and I actually have quite a bit of it, in this area), they need to be dedicated to a culture of Quality. And it is entirely possible for very talented engineers to produce shite. I've seen exactly that.

A culture of quality doesn’t require particularly skilled individuals to function. That’s in fact the thesis for the entire Deming management philosophy, and in line with what I’m saying (you can produce high quality with a good process or a good culture, you don’t necessarily need high caliber individuals)

Japan used a lot of Deming’s theories, to significant success. I worked for a Japanese company.

In my case, the company produced absolutely top-shelf stuff, but even relatively mediocre companies did well, using Deming’s techniques. It required that everyone be on board, wrt the culture, though.

But I have found that a “good” engineer is one that takes their vocation seriously. They may not be that accomplished or skilled, but they have self-discipline, humility, and structure.

I’ve met quite a few highly-skilled “not-good” engineers, in my day. I’m embarrassed to say that I’ve hired some of them.

Re: The Therac-25 Incident (2021)

#258

Post by bongodongobob »

simoncion wrote: post #254
bongodongobob wrote: post #189

Earlier quoted context omitted.

I have a microwave from the early 80s. If I stick a pencil in the door latch I can get it to run with the door open as well. It's not the demon core. Just don't stick your head in it when it's running.

> If I stick a pencil in the door latch I can get it to run with the door open as well. "The safety interlocks don't work when the operator intentionally goes out of his way to defeat them." isn't a concern. There's only so much you can do to prevent someone who's dedicated to disabling them. "The safety interlocks fail dangerous because of an unexpected power cut." is a huge concern. What else did the manufacturer s…

It's crazy to me that you are ruling out a power surge frying a board. Same thing could happen to the 80s model as well. You have not root caused it and are making up a failure mode that fits your point. Hell, the hall sensor could be fried and that's pretty damn mechanical. Again, your microwave isn't a demon core. Inverse square law applies. Don't put any limbs inside when it's on, and it really isn't that dangerous, so I'm not shocked they didn't apply aircraft safety design rules.

Re: The Therac-25 Incident (2021)

#259

Post by jonahx »

WalterBright wrote: post #208
benrutter wrote: post #5

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think t…

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the…

That makes sense. But wouldn't the "write quality software" and "test the software thoroughly" still be relevant to the individual pieces? If the chance of a catastrophic failure is the product of the failure rates of the pieces, getting P(PartFail) low helps too -- even if having multiple backups is the main source of protection.

Re: The Therac-25 Incident (2021)

#260

Post by simoncion »

bongodongobob wrote: post #258
simoncion wrote: post #254

Earlier quoted context omitted.

> If I stick a pencil in the door latch I can get it to run with the door open as well. "The safety interlocks don't work when the operator intentionally goes out of his way to defeat them." isn't a concern. There's only so much you can do to prevent someone who's dedicated to disabling them. "The safety interlocks fail dangerous because of an unexpected power cut." is a huge concern. What else did the manufacturer s…

It's crazy to me that you are ruling out a power surge frying a board. Same thing could happen to the 80s model as well. You have not root caused it and are making up a failure mode that fits your point. Hell, the hall sensor could be fried and that's pretty damn mechanical. Again, your microwave isn't a demon core. Inverse square law applies. Don't put any limbs inside when it's on, and it really isn't that dangerou…

> It's crazy to me that you are ruling out a power surge frying a board. Same thing could happen to the 80s model as well.

I'm taking OP at their word, in part because I don't feel like doing free-of-charge remote diagnostics on a microwave over a communications channel with such an absurdly high RTT. Based on your poor choice of counterexample, I was assuming that you didn't understand the difference between "as safe as we can reasonably make it" design and "it only appears to be safe" design (which is the worst kind).

> Don't put any limbs inside when it's on, and it really isn't that dangerous...

If true, then I'd wonder why they bother with any shielding at all. They'd save a bunch of money by putting a clear glass or plastic plate in front of the door, rather than all that metal, don'tcha think?

> Again, your microwave isn't a demon core.

Lots and lots of dangerous things are less dangerous than plutonium cores that are just raring to fizz a little. That doesn't mean that the safety mechanisms mandated to be incorporated in their design are obviously superfluous.

Post reply on HN