Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

171–180 of 307 posts

Re: The Therac-25 Incident (2021)

#171

Post by koverstreet »

A lot of people draw the wrong conclusions from Therac-25 today; becoming overly process driven can become a huge problem for software quality, because the processes have to be the right processes, and once processes are in place people have a natural tendency to defer to them and suspend their own judgement.

That gets actively dangerous; a lot of more recent safety mishaps are more of the variety of "processes were followed, but things went hilariously off the rails and no one noticed and spoke up".

Culture and expertise matter just as much if not more, especially today now that we all (in theory) should understand source control, testing, safer languages, etc.

I think Admiral Rickover's methods apply just as much today, and applying that kind of thinking would fill major gaps in a lot of organizations - he emphasized good communication, a sense of responsibility, and thinking on your feet, and his safety record is unmatched.

I think aviation also approaches process a bit better - by having much of it be more informal, less rigid checklists, it doesn't encourage people to suspend judgement so much.

There's also the Tankship Tromedy, which really emphasizes the engineering legwork of just chasing down, understanding and fixing every last failure mode you can find.

https://www.dieselduck.info/library/08%20policies/2006%20The...

Re: The Therac-25 Incident (2021)

#172

Post by bigstrat2003 »

edot wrote: post #65
greatgib wrote: post #60

Earlier quoted context omitted.

This story is kind of old. But also I'm suspicious that this was an AI generated content due to this weird paragraph (one becoming "they"): It's worth noting that there was one developer who wrote all of this code. They left AECL in 1986, and thankfully for them, no one has ever revealed their identity. And while it may be tempting to lay the blame at their feet—they made every technical choice, they coded every bug—…

Isn’t that the pronoun to use when you’re unsure of gender? This article didn’t feel AI-y to me.

The correct neutral pronoun in English is "he", though using "they" has been very popular in recent years due to a mistaken belief that it's sexist to use the male pronoun that way. I wouldn't say it's an AI smell at all.

Re: The Therac-25 Incident (2021)

#174

Post by mlnhd »

michaelt wrote: post #6

I'd be interested in knowing how many of y'all are being taught about this sort of thing in college ethics/safety/reliability classes. I was taught about this in engineering school, as part of a general engineering course also covering things like bathtub reliability curves and how to calculate the number of redundant cooling pumps a nuclear power plant needs. But it's a long time since I was in college. Is this sort…

This and Tacoma Narrows are literally the only topics covered in engineering ethics, which itself is literally only a one hour presentation.

Re: The Therac-25 Incident (2021)

#175

Post by dragonwriter »

bigstrat2003 wrote: post #172
edot wrote: post #65

Earlier quoted context omitted.

Isn’t that the pronoun to use when you’re unsure of gender? This article didn’t feel AI-y to me.

The correct neutral pronoun in English is "he", though using "they" has been very popular in recent years due to a mistaken belief that it's sexist to use the male pronoun that way. I wouldn't say it's an AI smell at all.

“They” has, in fact, been popular for gender neutral singular use since before the Latin-inspired prescriptivist movement and its attempts to install as “correct” various things that were previously either not dominant or in some cases entirely foreign to English; that movement's influence had actually been fading on that usage and in general for decades even before concerns about sexism in language became significant; singular “they” is no more incorrect than the now-so-universal-most-have-forgotten-the-actual-singular use of the second-person plural when the semantics are in fact singular.

Re: The Therac-25 Incident (2021)

#176

Post by superjan »

elric wrote: post #43

One of the commenters on the article wrote this: > Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. I was briefly part of an experiment with electronic patient records in an ICU in the early 2000s. My job was to basically babysit the server processing the records in…

It”s worthwhile to mention that in the US and EU EMRs are generally not considered Medical Devices and are therefore not subject to a lot of regulations.

https://www.medicaleconomics.com/view/what-if-emrs-were-clas...

Re: The Therac-25 Incident (2021)

#177

Post by cantrevealname »

> All of this software, from the individual processes to the OS itself, were the work of a single software developer. They left AECL in 1986, and no one has ever revealed their identity.

I bet some readers are thinking that the developer that caused this tragedy retired with the millions he earned, maybe sailed his yacht to his Caribbean mansion. But the $300K FAANG salaries and multi-million stock options for senior developers represents the last decade or two. In the 1980's, developers were paid poorly and commanded little respect. The heroes in tech companies that sold expensive devices were the salesmen back then. The commission on the sale of a single Therac-25 probably exceeded the developer's salary.

All of the following would indicate that this developer, no matter how senior or capable, was still a low-paid schlub:

- It's Canada, so automatically 20% lower salaries than in the U.S. (AECL is in Canada, so it's a good bet that the developer was Canadian.)

- It's the 1980s, so pre-web, pre-smartphones, pre-Google/Amazon, and developers had little recognition and low demand.

- It's government, known to pay poorly for developers. (AECL is a government-owned corporation.)

- It's mostly embedded software. Even though embedded software can be incredibly complex and life-critical, it's the least visible, so it's among the lower paid areas of software engineering (even today).

For 1986, I would put his salary at $30-50K Canadian, or converted to U.S. dollars at that time would be $26-43K U.S., and inflation adjusted would be $78-129K U.S. today. And no stock options.

Re: The Therac-25 Incident (2021)

#178

Post by HankStallone »

greatgib wrote: post #60
mellosouls wrote: post #24

TIL TheDailyWTF is still active. I'd thought it had settled to greatest hits only some years ago.

This story is kind of old. But also I'm suspicious that this was an AI generated content due to this weird paragraph (one becoming "they"): It's worth noting that there was one developer who wrote all of this code. They left AECL in 1986, and thankfully for them, no one has ever revealed their identity. And while it may be tempting to lay the blame at their feet—they made every technical choice, they coded every bug—…

It writes that way because almost everyone writes that way these days. It's annoying if you learned English grammar from textbooks and other materials written over 50 years ago, but it's extremely common now anyway. So a large chunk of its training data will be that way.

It's interesting, because all the older works in its training data will default to the masculine singular, and that has to be a massive number of books too. But maybe the modern writing, including lots of online sources, simply overwhelms that. Or it's one of the guardrails written into the AIs to avoid offending people.

Re: The Therac-25 Incident (2021)

#180

Post by MarkusWandel »

In a quick skim of the comments so far, I don't see the real smoking gun.

The previous devices had hardware interlocks. So if the software glitched, it was just an annoying glitch - nobody got zapped. But mature software gets trusted, so they removed the hardware interlock as redundant. And then the annoying glitches became fatal. Total miscommunication. The people cost-reducing the hardware interlock only saw mature, trustworthy software. The people living with the glitches only saw them as annoying, but harmless. And then, disaster.

Post reply on HN