Worst Computer Bugs in History: Therac-25 (2017)
blog.bugsnag.com
Worst Computer Bugs in History: Therac-25 (2017)
1–10 of 114 posts
Re: Worst Computer Bugs in History: Therac-25 (2017)
#2A descriptive video can be found here: https://www.youtube.com/watch?v=uEvu2PlDhO0
Re: Worst Computer Bugs in History: Therac-25 (2017)
#3How horrible it must have been for the operator, to realize they had killed two patients, through no fault of their own.
Re: Worst Computer Bugs in History: Therac-25 (2017)
#4The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code.
Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.
Re: Worst Computer Bugs in History: Therac-25 (2017)
#5Therac is one of the reasons I get nervous about "health hacking." Yes, people can verifiably benefit from some of the advancements made in this movement, like the DIY diabetic insulin pump, and yes, I prefer to see such advancements be open source than locked up in proprietary designs and trade secrets. And there probably is room in health regulation for trimming the red tape anyway even for innovations originating from the commercial sector.
On the other hand, when corners are cut (no hardware interlocks, for example) and edge cases aren't considered, even innocently, then you get things like this. It makes products more expensive to design and more costly to buy and maintain to do the extra engineering. It is certainly a barrier to entry, too. But do we want another case like this because people said "this is good enough"?
Re: Worst Computer Bugs in History: Therac-25 (2017)
#6As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...) from what was also a fairly subtle engineering failure.
IMO, harm from software bugs (so far) have been vastly surpassed by explicit choices in system design. The various emission cheat scandals have almost certainly taken a real toll on human life going into the hundreds of person lives. More subtly, the choices to retain data inappropriately at Ashley Madison (probably) lead directly to suicides and serious emotional harm. Those are just the two recent examples that spring to mind as a practocing developer, not an ethicist.
To somewhat over simplify but when discussing engineering ethics, the harm from software developers building things wrong is swamped by building the wrong things.
Re: Worst Computer Bugs in History: Therac-25 (2017)
#7The Therac-25 is part of the core curriculum in computer engineering, but I wonder if it's actually (in the grand scheme of things) hat bad of an incident. Compared with Facebook fomenting ethnic cleansing in Asia, the people who were hurt or died were very limited. Are there any new(er) examples which can show the dangers of a widely distributed, connected horror?
Re: Worst Computer Bugs in History: Therac-25 (2017)
#8The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.
Lawful punishment of bad quality software needs to be a thing, just like in other industries.
Only then will most companies actually start to care about software quality in their development processes.
Re: Worst Computer Bugs in History: Therac-25 (2017)
#9I read this article, and many years ago the full report, and one of the omissions on the list of causes that stood out to me was overcomplexity --- if you read about the possible functions of the machine, they really don't require multiple threads much less a full multitasking OS. None of these race conditions would've occurred if it was a simple single-threaded embedded controller.
To paraphrase an old Hoare quote, software can either be so simple it obviously contains no bugs, or so complex that it contains no obvious bugs.
Re: Worst Computer Bugs in History: Therac-25 (2017)
#10According to the Wikipedia entry on the Therac-25, it was "In response to incidents like those that the IEC 62304 standard was created, which introduces development life cycle standards for medical device software and specific guidance on using software of unknown pedigree".
For those working in safety and quality control of medical systems, how much does compliance to those specifications actually diminish the chances of another Therac-25 incident?
Considering that automation continues to increase, from automatic patient table positioning up to diagnose-assisted AI, are there new challenges when it comes to designing medical systems in order to keep them safe and maintainable? How likely is it for the FDA or the equivalent agencies around the globe to authorize the use of open source systems?