Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Therac-25 Incident (2021)

thedailywtf.com

Post reply on HN
Print view View original

41–50 of 307 posts

Re: The Therac-25 Incident (2021)

#41

Post by SCdF »

isopede wrote: post #9

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…

The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more.

The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

Re: The Therac-25 Incident (2021)

#42

Post by NitpickLawyer »

solids wrote: post #13
NitpickLawyer wrote: post #7

Earlier quoted context omitted.

The MCAS related bugs @ Boeing led to 300+ deaths, so it's probably a contender.

Was that a bug or a failure to inform pilots about a new system?

I would say plenty of both. They obviously had to inform the pilots, but the way the system didn't reset permanently after 2-3 (whatever) sessions of "oh, the pilot trimmed manually, after 10 seconds we keep doing the same thing" was a major major logic blunder. Failure all across the board, if only from the perspective of end-to-end / integration testing if nothing else.

Worryingly, e2e / full integration testing was also the main cause of other Boeing blunders, like the Starliner capsule.

Re: The Therac-25 Incident (2021)

#43

Post by elric »

One of the commenters on the article wrote this:

> Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper.

I was briefly part of an experiment with electronic patient records in an ICU in the early 2000s. My job was to basically babysit the server processing the records in the ICU.

The entire staff hated the system. They hated having to switch to computers (this was many years pre-ipad and similarly sleek tablets) to check and update records. They were very much used to writing medications (what, when, which dose, etc) onto bedside charts, which were very easy to consult and very easy to update. Any kind of dataloss in those records could have fatal consequences. Any delay in getting to the information could be bad.

This was *not* just a case of doctors having unfounded "feelings" that computers were dangerous. Computers were very much more dangerous than pen and paper.

I haven't been involved in that industry since then, and I imagine things have gotten better since, but still worth keeping in mind.

Re: The Therac-25 Incident (2021)

#45

Post by mdavid626 »

Some sanity checks are always a good idea before running such destructive action (IF beam_strength > REASONABLY_HIGH_NUMBER THEN error). Of course the UI bug is hard to catch, but the sanity check would have prevented this completely and the machine would just end up in an error, rather than killing patients.

Re: The Therac-25 Incident (2021)

#46

Post by A1kmm »

napolux wrote: post #3

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

Not even close. Israel apparently has AI bombing target intel & selection systems called Gospel and Lavender - https://www.theguardian.com/world/2024/apr/03/israel-gaza-ai.... Claims are these systems have a selectivity of 90% per bombing, and they were willing to bomb up to 20 civilians per person classified by the system as a Hamas member. So assuming that is true, 90% of the time, they kill one Hamas member, and up to 20 innocents. 10% of the time, they kill up to 21 innocents and no Hamas members.

Killing 20 innocents and one Hamas member is not a bug - it is callous, but that's a policy decision and the software working as intended. But when it is a false positive (10% of the time), due to inadequate / outdated data and inadequate models, that could reasonably classified as a bug - so all 21 deaths for each of those bombings would count as deaths caused by a bug. Apparently (at least earlier versions) of Gospel were trained on positive examples that mean someone is a member of Hamas, but not on negative examples; other problems could be due to, for example, insufficient data, and interpolation outside the valid range (e.g. using pre-war data about, e.g. how quickly cell phones are traded, or people movements, when behaviour is different post-war).

I'd therefore estimate that deaths due to classification errors from those systems is likely in the thousands (out of the 60k+ Palestinian deaths in the conflict). Therac-25's bugs caused 6 deaths for comparison.

Re: The Therac-25 Incident (2021)

#47

Post by b_e_n_t_o_n »

mdavid626 wrote: post #45

Some sanity checks are always a good idea before running such destructive action (IF beam_strength > REASONABLY_HIGH_NUMBER THEN error). Of course the UI bug is hard to catch, but the sanity check would have prevented this completely and the machine would just end up in an error, rather than killing patients.

invariants are so useful to enforce even for toy projects. they should never be triggered outside of dev, but if they do sometimes it's better to just let it crash.

Re: The Therac-25 Incident (2021)

#48

Post by danadam »

napolux wrote: post #3

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

Some Google Pixel phones couldn't dial emergency number (still can't?). I don't know if there were any deadly consequences of that.

https://www.androidauthority.com/psa-google-pixel-911-emerge...

Re: The Therac-25 Incident (2021)

#49

Post by hahn-kev »

SCdF wrote: post #41
isopede wrote: post #9

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…

The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.

Re: The Therac-25 Incident (2021)

#50

Post by b_e_n_t_o_n »

echelon wrote: post #11
napolux wrote: post #3

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

The 737 Max MCAS is arguably a bug. That killed 346 people. Not a "bug" per se, but texting while driving kills ~400 people per year in the US. It's a bug at some level of granularity. To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up.

> To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up.

These kind of calculations always make me wonder...say someone wasted one minute of everybody's life, is the cost ~250 lives? One minute? Somewhere in between?

Post reply on HN