Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Worst Computer Bugs in History: Therac-25 (2017)

blog.bugsnag.com

Post reply on HN
Print view View original

11–20 of 114 posts

Re: Worst Computer Bugs in History: Therac-25 (2017)

#11

Post by btrettel »

topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing.

Of course, as you have said, building the wrong thing swamps other harms. Unintended consequences are hard to predict, unfortunately, but I am interested in ways to improve this situation. Standards design also interests me, particularly standards which are hard to cheat.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#12

Post by zzzcpan »

pjmlp wrote: post #8
hedora wrote: post #4

The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.

Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.

I don't think it works, at least not within the current legal system. Where it becomes mostly about legal bureaucracy of avoiding responsibility, rather than truly focusing on reliability.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#13

Post by jsjohnst »

LeoPanthera wrote: post #3

How horrible it must have been for the operator, to realize they had killed two patients, through no fault of their own.

Honestly, I disagree slightly. Reading the article as well as the original report years ago, I wasn’t left with the feeling the operator made “no fault of their own”. Are they to blame at all, no, but the operator certainly made mistakes. For example, assuming an error is innocuous when you are intentionally delivering radiation to a person is careless at best. Again, the machine is at fault solely, but that doesn’t mean the operator didn’t have a role in the death.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#14

Post by azernik »

btrettel wrote: post #11
topkai22 wrote: post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#15

Post by btrettel »

azernik wrote: post #14
btrettel wrote: post #11

Earlier quoted context omitted.

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

Do you say that to give poor software engineering legal consequences?

Re: Worst Computer Bugs in History: Therac-25 (2017)

#16

Post by pjmlp »

zzzcpan wrote: post #12
pjmlp wrote: post #8

Earlier quoted context omitted.

Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.

I don't think it works, at least not within the current legal system. Where it becomes mostly about legal bureaucracy of avoiding responsibility, rather than truly focusing on reliability.

Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box.

The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back.

Also every time that there is an bunch of black hat hackers that expose company internal data, if the security breach can be mapped into a CVE database entry, a good law firm could probably make something out of it.

Not all jurisdictions are alike, but one needs to start somewhere.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#17

Post by pjmlp »

azernik wrote: post #14
btrettel wrote: post #11

Earlier quoted context omitted.

Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have sai…

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

In Portugal we do have it, and while it is optional to join it if you don't legally sign off projects on the company's name, at least they certify that University degrees are actually teaching proper software engineering.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#18

Post by chabsf »

btrettel wrote: post #15
azernik wrote: post #14

Earlier quoted context omitted.

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

Do you say that to give poor software engineering legal consequences?

In Canada, that's the legal definition of engineering. You may not call yourself an engineer without accreditation and such accreditation will be rescinded if you make severe enough engineering mistakes.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#20

Post by _jal »

btrettel wrote: post #15
azernik wrote: post #14

Earlier quoted context omitted.

The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.

Do you say that to give poor software engineering legal consequences?

Not the poster, but I think there are multiple paths of action encouraged by a code similar to other engineering disciplines[1].

More involvement of the legal and insurance industries are one of them. Another is to give software engineers something solid to brace themselves on when pushing back at management - completely aside from consequences for the company, if you're bonded or worry about a license, there are some things you won't let your manager sweet-talk you in to. Another is to provide a model of behavior for engineers, like it says on the tin. It doesn't mean everyone will, or even that the model is always absolutely correct. But giving folks a way to think about things when they feel something's off is a good thing.

Yet another is theoretically providing a baseline of competence. I think that depends more on improving informal culture than any formal mechanism, though.

[1] Note: not arguing in favor of one; I haven't made up my mind on what I think about the topic.

Post reply on HN