Earlier quoted context omitted.
I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.
Right, but if everyone is low quality then there's no one to do that correction. That may seem a bit hypothetical but it can easily happen if you have a company that systematically underpays, which I'm sure many of us don't need to think hard to imagine, in which case they will systematically hire poor developers (because those are the only ones that ever applied).
The Therac-25 Incident (2021)
71–80 of 307 posts
Re: The Therac-25 Incident (2021)
#72I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…
The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.
In the Therac-25 case, the killing was quite immediate and it would have happened even if the correct radiation dose was recorded.
Re: The Therac-25 Incident (2021)
#73Earlier quoted context omitted.
Was that a bug or a failure to inform pilots about a new system?
Not a bug. A non airworthy plane they tried to patch up with software.
[edit as I can't reply to the child comment]: The FAA and EASA both looked into the stall characteristics afterwards and concluded that the plane was stable enough to be certified without MCAS and while it did have more of a tenancy to pitch up at high angles of attack it was still an acceptable amount.
Re: The Therac-25 Incident (2021)
#74Well There's Your Problem podcast, Episode 121: Therac-25 https://www.youtube.com/watch?v=7EQT1gVsE6I
Re: The Therac-25 Incident (2021)
#75>One failure occurred when a particular sequence of keystrokes was entered on the VT100 terminal that controlled the PDP-11 computer: If the operator were to press "X" to (erroneously) select 25 MeV photon mode, then use "cursor up" to edit the input to "E" to (correctly) select 25 MeV Electron mode, then "Enter", all within eight seconds of the first keypress and well within the capability of an experienced user of the machine, the edit would not be processed and an overdose could be administered. These edits were not noticed as it would take 8 seconds for startup, so it would go with the default setup
Kinda reminds me how everything is touchscreen nowadays from car interfaces to industry critical software
Re: The Therac-25 Incident (2021)
#76Earlier quoted context omitted.
Not a bug. A non airworthy plane they tried to patch up with software.
The plane was perfectly airworthy without MCAS, that was never the issue. The issue was it handled differently enough at high angles of attack to the 737NG that pilots would've needed additional training or possibly a new type rating without MCAS changing the trim in this situation. The competition (Airbus NEO family) did not need this kind of new training for existing pilots, so airlines being required to do this fo…
Re: The Therac-25 Incident (2021)
#77Re: The Therac-25 Incident (2021)
#78I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a…
talk to anyone in the industries about 'automation' on medical or critical infra devices and they will tell you NO. No touching our devices with your rubbish. i am pretty confident they wont let claude touch if it they dont even let deterministic automations run... that being said, maybe there are places. but this is always the sentiment i got. no automating, no scanning, no patching. device is delivered certified an…
I don't have the same faith in corporate leadership as you, at least not when they see potentially huge savings by firing some of the expensive developers and using AI to write more of the code.
Re: The Therac-25 Incident (2021)
#79TIL TheDailyWTF is still active. I'd thought it had settled to greatest hits only some years ago.
This story is kind of old. But also I'm suspicious that this was an AI generated content due to this weird paragraph (one becoming "they"): It's worth noting that there was one developer who wrote all of this code. They left AECL in 1986, and thankfully for them, no one has ever revealed their identity. And while it may be tempting to lay the blame at their feet—they made every technical choice, they coded every bug—…
Re: The Therac-25 Incident (2021)
#80What surprised me most was that only one developer was working on such an unpredictable technology, whereas I think I need at least 5 developers to be able to discuss options.
BTW: Relevant XKCD: https://xkcd.com/2347/