Worst Computer Bugs in History: Therac-25 (2017) • Hacker News Board

Re: Worst Computer Bugs in History: Therac-25 (2017)

#21

Post by jacquesm » Sat, Aug 11, 2018, 6:12 PM UTC

classichasclass wrote: post #5

Therac is one of the reasons I get nervous about "health hacking." Yes, people can verifiably benefit from some of the advancements made in this movement, like the DIY diabetic insulin pump, and yes, I prefer to see such advancements be open source than locked up in proprietary designs and trade secrets. And there probably is room in health regulation for trimming the red tape anyway even for innovations originating…

I posted a link about a insulin pump that could be hacked into remotely. Even if it isn't 'health hacking' because you've bought a product from what you thought was a reputable vendor there are no guarantees that it will be secure and bug free.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#22

Post by anonuser123456 » Sat, Aug 11, 2018, 6:13 PM UTC

pjmlp wrote: post #16

zzzcpan wrote: post #12

Earlier quoted context omitted.

I don't think it works, at least not within the current legal system. Where it becomes mostly about legal bureaucracy of avoiding responsibility, rather than truly focusing on reliability.

Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the secur…

Probably not. As a company you just disclaim liability in your terms of service.

Jurisdictions that try and override this, simply get excluded from the customer base.

The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#23

Post by pjmlp » Sat, Aug 11, 2018, 6:14 PM UTC

dangom wrote: post #10

According to the Wikipedia entry on the Therac-25, it was "In response to incidents like those that the IEC 62304 standard was created, which introduces development life cycle standards for medical device software and specific guidance on using software of unknown pedigree". For those working in safety and quality control of medical systems, how much does compliance to those specifications actually diminish the chanc…

> How likely is it for the FDA or the equivalent agencies around the globe to authorize the use of open source systems?

Actually they already authorize stuff like Qt.

Computer systems where human lives are put in risk belong to what is called High Integrity Computing.

There are very strict coding standards, where even C looks more like Ada than proper C.

https://ldra.com/medical/

https://www.qt.io/qt-in-medical/

https://www.vectorcast.com/testing-solutions/software-testin...

Source code availability is not an issue, because it is part of the certification process to provide it.

The problem is having the money to pay for a certification, which becomes invalid the moment anything gets changed, namely compiler being used, source code, or if any of the third party dependencies gets updated.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#24

Post by fedorareis » Sat, Aug 11, 2018, 6:15 PM UTC

azernik wrote: post #14

btrettel wrote: post #11

Earlier quoted context omitted.

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

May I point you to the ACM/IEEE-CS Software Engineering Code of Ethics https://ethics.acm.org/code-of-ethics/software-engineering-c... this was a major thing discussed in my professional ethics corse in college.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#25

Post by nv-vn » Sat, Aug 11, 2018, 6:15 PM UTC

jsjohnst wrote: post #13

LeoPanthera wrote: post #3

How horrible it must have been for the operator, to realize they had killed two patients, through no fault of their own.

Honestly, I disagree slightly. Reading the article as well as the original report years ago, I wasn’t left with the feeling the operator made “no fault of their own”. Are they to blame at all, no, but the operator certainly made mistakes. For example, assuming an error is innocuous when you are intentionally delivering radiation to a person is careless at best. Again, the machine is at fault solely, but that doesn’t…

Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if they don't respond appropriately to an error.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#26

Post by hexane360 » Sat, Aug 11, 2018, 6:16 PM UTC

hedora wrote: post #4

The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.

I don't think you can say the lethal bugs were all in the frontend code. First, machines of that age didn't have as clear of a distinction between front and backend. Second, any good back end has bulletproof safeguards against bad frontend input. It makes more design sense to safeguard the backend against spamming input than the frontend, because the frontend is more likely to require redesign, and multiple frontends can interact with the same backend.

More than anything else, this accident shows the importance of fuzz testing your critical logic, the importance of hardware interlocks, and the importance of multiple independent layers of interlocks.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#27

Post by astraltheman » Sat, Aug 11, 2018, 6:19 PM UTC

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

Software is honestly cheaper and easier to test, other kinds of engineering tests run far more expensive in time and materials, and simulations aren't perfect and can't replicate all real world conditions.

Definitely agree on the explicitly bad choices though, and since software's impact is often very subtle it might really be impossible to gauge exactly how bad some of those choices end up being.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#28

Post by walrus01 » Sat, Aug 11, 2018, 6:22 PM UTC

azernik wrote: post #14

btrettel wrote: post #11

Earlier quoted context omitted.

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

I am personally more concerned with software engineers and network engineers aiding and abetting the imprisonment, torture and execution of people by repressive regimes, by enabling surveillance technology and fucking with internet traffic analysis. Way more people are going to be hurt in the near term by that than by therac-25 type mistakes.

For example if you're a Chinese network engineer, and you can avoid it, don't take a job setting up tracking and database of Uyghur people. That is an ethical issue just as important as the therac-25 type problem.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#29

Post by jacquesm » Sat, Aug 11, 2018, 6:23 PM UTC

pjmlp wrote: post #16

zzzcpan wrote: post #12

Earlier quoted context omitted.

I don't think it works, at least not within the current legal system. Where it becomes mostly about legal bureaucracy of avoiding responsibility, rather than truly focusing on reliability.

Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the secur…

It's a huge case of Stockholm Syndrome, end users have been conditioned over years to accept these things as normal and have become engaged in an abusive relationship with their captors who will withhold the little help they are prepared to give if the users dare to complain.

No other industry has ever gotten away with this. But with 'software eating the world' change is just around the corner, the first software bug that will kill a few thousand people will be a very rude wake up call that something needs to be done.

The only industry that really gets it is aviation, medical tries hard but is still a mess, with the exception of devices, in general those are engineered reasonably well.

In a way all these SaaS products are setting the stage for some real liability, after all, if the end user doesn't have even a modicum of control over what happens with their data then the other party should assume liability, even if they try real hard to disclaim that.

Open source might get exempted, if not then I suspect that a lot of open source projects will fold.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#30

Post by walrus01 » Sat, Aug 11, 2018, 6:25 PM UTC

chabsf wrote: post #18

btrettel wrote: post #15

Earlier quoted context omitted.

Do you say that to give poor software engineering legal consequences?

In Canada, that's the legal definition of engineering. You may not call yourself an engineer without accreditation and such accreditation will be rescinded if you make severe enough engineering mistakes.

I don't disagree with you, but there's a LOT of people in Canada calling themselves software engineers or network engineers who don't have a degree qualified to wear the iron ring.