Live data from Hacker News

Worst Computer Bugs in History: Therac-25 (2017)

blog.bugsnag.com

81–90 of 114 posts

Re: Worst Computer Bugs in History: Therac-25 (2017)

#81
post #79

Earlier quoted context omitted.

No because it was also caused by an engineer, so this could lead to terrifying incentives.

How is this different from rewarding a salesman who rescues a sale that a different salesman had botched?

Because you can't purposefully botch a sale in order to later recover it. Also because you can't avoid botching sales by being more conservative or adding more process. In short, sales and engineering have basically nothing in common.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#82

Earlier quoted context omitted.

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement. When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated. I asked why he does not wire the switches to cut power directly to be on the safe side. His answer "It's to much bo…

...but it's not "much bother" to run the control loop through the complexity of software? This seems to be the equivalent of overengineering in software, where something straightforward is instead performed through many layers of abstraction and indirection. The straightforward way of implementing this with a bidirectional motor is to wire normally-closed limit switches in series with their appropriate direction sign…

Try that with stepper motors used in 3d printers. Try it with an off the shelf driver. Make it so the limit switch only stops motion in one direction.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#83
post #44

Earlier quoted context omitted.

Probably not. As a company you just disclaim liability in your terms of service. Jurisdictions that try and override this, simply get excluded from the customer base. The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.

>Probably not. As a company you just disclaim liability in your terms of service. Thankfully EULAs are void in Europe. It is all a matter how big the customer base gets, I am hoping eventually we get something like that EU wide. > The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out. If that was true 1 € shops wouldn't exist, but even those products have mor…

> Thankfully EULAs are void in Europe.

It's not so clear cut that you should be thankful. The ability of companies to dictate the terms of which users can use their software, affects their risk calculation to produce the product in the first place. It is very likely that very useful but imperfect software will not be written because the risk / reward balance is tilted.

Remember, you always have the ability to reject an EULA; simply don't use the product.

> If that was true 1 € shops wouldn't exist, but even those products have more testing than most software out there.

Consumers can make value choices on quality vs cost. This is a basic market function.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#84
post #8
post #4

The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.

Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.

Other industries are also free to make terrible products, they're just not allowed to hurt people. In that regard, software isn't that different.

Crappy software just doesn't physically injure people very often (compared to like, lawnmowers), and that's where the most serious legal liability for products comes from. Monetary damage from software gets worked out the same way any contract dispute gets worked out, or the same way a physical product that doesn't work but doesn't hurt anyone would get worked out.

Liability for bad software is also complicated by the fact that there's a million apps out there that are free to use. If they are broken for however long, it's hard to say that it cause monetary damage to anyone. (If anything, people are saving time... to paraphrase Mitch Hedberg, FB is broken, sorry for the convenience.)

Re: Worst Computer Bugs in History: Therac-25 (2017)

#85
post #6

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap... ) from what was also a fairly subtle engineering failure…

We’re already seeing bugs kill or maim with autonomous cars. You can be sure there is far worse data associated with military system flaws that we don’t know about.

In any case, it’s dangerous to explain away software failure by dumping blame on systems. The lack of professional standards in software makes it easy for people to do bad things well.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#86
post #35

One of my favorite software horror stories is the one of $32 _billion_ overdraft by the Bank of New York. From "Computer-Related Risks" by Peter G. Neumann, published 1994 (REALLY recommended reading) "One of the most dramatic examples was the $32 billion overdraft experienced by the Bank of New York (BoNY) as the result of the overflow of a 16-bit counter that went unchecked. (Most of the other counters were 32-bits…

Imaging being called in. Ok guys we don't know what the problem is but it's costing the company $3500 per minute the bug stays unfixed just in interests. No pressure.

I was on a conference call with a bank that due to a number miscommunications and general idiocy was under the impression that they were possibly in violation of some banking laws (no comment on what laws) because they misinterpreted what exactly was happening. Their lawyers supposedly told them they were at risk for an unbelievable sum of fines and criminal stuff.

In short they saw some data they didn't understand that seemed to indicate things at the bank were very much not how they were thought to be, and indicated something very specific was happening. In reality that was not the case but a lot of assumptions by morons somehow were believed and everything snowballed into lots of corroborating "evidence" that seemed to indicate bad things. This was a bank that rarely had this level of stupid information and assumptions rise to the top so nobody actually questioned it no matter how absurd it seemed.

My presence on the call was purely because the bank wanted every vendor they had looking to see if they saw problems, so I was basically just poking around telling them what i saw from my end.

At one point on the call there was a dude in an unmanned data center literally just flipping off power switches and cutting the cables of various equipment as he was given the locations for it. It was frantic and very unamerican bank like (at least the ones I worked with were pretty cool customers normally). I turned my speakerphone on to let my coworkers listen to the chaos.

In the end, it was a stupid SQL server related virus thing that created some good old fashioned network disruption and when that was solved and nothing looked like the sky was falling, everyone came to their senses. Once the dude pulled the power to enough of them everything calmed down and I checked my bank account and it was all there... but no excess either :(

Re: Worst Computer Bugs in History: Therac-25 (2017)

#87
post #79

Earlier quoted context omitted.

How is this different from rewarding a salesman who rescues a sale that a different salesman had botched?

Because you can't purposefully botch a sale in order to later recover it. Also because you can't avoid botching sales by being more conservative or adding more process. In short, sales and engineering have basically nothing in common.

I don't think GP was talking about a scenario where the same engineer who created the bug fixed it and gets rewarded, rather one where a different engineer fixes it. Of course it wouldn't make sense as you describe it.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#88
post #18

Earlier quoted context omitted.

Do you say that to give poor software engineering legal consequences?

In Canada, that's the legal definition of engineering. You may not call yourself an engineer without accreditation and such accreditation will be rescinded if you make severe enough engineering mistakes.

That's about being a Professional Engineer, which legally entitles you to certain actions (e.g. sign off on design docs). Anyone can call themselves engineers of any kind as long as they don't pretend to be P.Engs. I am not sure though that being P.Engs in software means anything in practice, unlike in e.g. structural engineering.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#89
post #54

Earlier quoted context omitted.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked. Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf : "In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous ma…

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement. When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated. I asked why he does not wire the switches to cut power directly to be on the safe side. His answer "It's to much bo…

While it's not terribly uncommon for small hobby machines to depend on software limits I can't think of a single instance in my years of maintaining "real" production CNC machines of encountering a machine that didn't also include hard limits.

Limit switches typically include two trip points. The first is monitored by the control system; when it is tripped the control halts execution and stops the machine. The second limit is wired directly to the servo amplifier so that, if for what ever reason, the control fails to halt the machine when the soft limit is tripped power is removed and motion is halted. Both limits are fail-safe such that if they were to become disconnected it would result in a limit exceeded condition.

Re: Worst Computer Bugs in History: Therac-25 (2017)

#90
post #54

Earlier quoted context omitted.

In my opinion, one of the most tragic aspects of these horrific incidents is that the predecessors of the Therac-25 actually had independent protective circuits and other measures to ensure safe operations, which the Therac-25 lacked. Here is a quote from http://sunnyday.mit.edu/papers/therac.pdf : "In addition, the Therac-25 software has more responsibility for maintaining safety than the software in the previous ma…

I have friend who has 40 years programming experience, he is building a computer controlled milling machine in his basement. When I asked him about the limit switches it turns out they are read by software only and the software will turn off power to the motor controllers if a limit switch is activated. I asked why he does not wire the switches to cut power directly to be on the safe side. His answer "It's to much bo…

This seems like a good time to mention that God gave us hardware interrupt inputs and watchdog timers.
Post reply on HN