Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994) • Hacker News Board

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#31

Post by kalleboo » Sun, Jul 06, 2014, 12:25 PM UTC

lstamour wrote: post #30

blabby wrote: post #27

At the time of Therac-25, FDA was only budgeted to investigate 6 percent of device applications. Currently, the same mistakes made in the eighties with Therac-25 are being made in many radiation therapy devices. The two NY Times articles (Pulitzer Prize winning) in 2010 and 2011 describe some of the newer cases. What's shocking to me is that the incidents are always reported in isolation. People become briefly outrag…

Why hasn't the hardware failsafe for overdoses become mandatory? Why don't we apply defense-in-depth to all worst-case scenarios involving deadly things? Of course, sometimes hospitals aren't logical, air circulation between rooms comes to mind. And here, I'm sure everyone just trusts the machines because they paid a lot of money for them and it's always worked in the past ...

> Why hasn't the hardware failsafe for overdoses become mandatory? Why don't we apply defense-in-depth to all worst-case scenarios involving deadly things?

Because money

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#32

Post by noir_lord » Sun, Jul 06, 2014, 1:01 PM UTC

kohanz wrote: post #2

The Therac-25 case study is a tragic one, but fortunately it is not forgotten. I work on medical devices (and have worked on radiotherapy devices previously) and the standards for quality systems and regulatory hurdles (which I occasionally see bemoaned here on HN) are there with good reason. In fact, Therac-25 is often cited when training new hires on quality (as required with any ISO-13485 compliant QMS).

Definitely not forgotten, I wrote an essay on software safety when I was at college (not Uni, UK meaning) and the Therac 25 was a big chunk of it.

The research I did has always stuck with me because of the suffering the patients where exposed to combined with a company attitude of "admit only what we are forced to so we don't hurt sales".

Nothing I write is safety critical (though it was a field I was fascinated with when I was younger) though so I can sleep at night.

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#33

Post by merraksh » Sun, Jul 06, 2014, 1:37 PM UTC

pathdependent wrote: post #23

merraksh wrote: post #20

The Therac-25's software program, relatively crude by today's standards, probably contained 101000 lines of code. At one error for every 500 lines, that works out to the possibility of twenty errors. I'd say 200, not twenty.

I think the article was OCR'd. There were a few other mistakes that were clearly misinterpreted characters. I think the 1 in the thousands place is actually a comma on the source.

Good point. There are many hints that article was OCR'd, for instance an ".4ECL" instead of "AECL".

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#34

Post by JoshTheGeek » Sun, Jul 06, 2014, 4:51 PM UTC

101000 lines, 500 lines per error gives about 200 errors, not 20

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#35

Post by thaumasiotes » Sun, Jul 06, 2014, 4:53 PM UTC

gnaffle wrote: post #3

Thanks, I hadn't read this before. For those that haven't read it, here's Levesons article on the Therac-25: http://sunnyday.mit.edu/papers/therac.pdf

This one made me think about public outrage against tobacco companies.

One minor theme in this article is that AECL denied knowledge of any reports of Therac-25 malfunctions even when, looking at a timeline of publicly-known events, such ignorance might be described as "implausible".

They don't seem to have been punished for this, and while I agree that it isn't laudable I also agree that it's not the greatest infraction. AECL really did care about the proper functioning of their machine. They really did look for problems. They cooperated with the FDA to a very great extent. It's hard to fault them for not thinking of testing "what if we enter incorrect configuration information, and then correct it within 8 seconds?"

But tobacco companies are routinely vilified for sitting on cigarette mortality data, as if this was by itself enough to make them irredeemable. They didn't even get off with a light punishment, much less the zero punishment AECL received. I suspect the difference, in the minds of many, is that AECL was a benign company advancing a useful purpose, while tobacco companies sold a product whose only use was to kill the operator. But that was legal then and remains legal today -- how can it be the justification for punishing them extra-hard for otherwise minor problems? AECL's misrepresentedly-unsafe product didn't even kill the operator; it killed random sick people who trusted the hospitals.

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#36

Post by mbzastava » Sun, Jul 06, 2014, 4:56 PM UTC

I remember this case study being taught to us at Engineering Ethics 101 while I was in school for my BSEE. I clearly recall the question posed to the room of about 50 would-be electrical engineers. In the case study it was defined that the engineer working on the software was different than the engineer working on the hardware. The class was to take the hardware engineer's perspective for the exercise. The question was "WHO IS RESPONSIBLE?"

Never mind the gravity of such a question being posed to a 18 year-old fresh out of high school. What shocked me was the professor's answer "THE SOFTWARE ENGINEER!"

I'm no programmer. I nearly failed to achieve my BSEE due to the required programming courses. But never in my life would I imagine to rationalize this type of blame onto someone else, let alone have it taught to me in such a point-blank manner.

I raised my hand and offered my objection. "HOW CAN YOU BLAME THE SOFTWARE ENGINEER, WHEN YOU DESIGNED THE MACHINE THAT WAS CAPABLE OF GENERATING ENOUGH RADIATION TO KILL A PERSON?" I further objected "YOU ALSO LEFT IT WITHOUT ANY HARDWARE-BASED FAILSAFE MEASURES, RELYING SOLELY ON THE SOFTWARE PERFORMANCE."

Dead silence from the class as the professor responded in a quip "NO. IT IS THE SOFTWARE ENGINEER'S FAULT. MOVING ON..."

EDIT: We had been given a photo-copy of an article on the accidents for the exercise. No where did I recall reading they had implemented a solution (this was in 2001) that was primarily hardware based. I would have loved to shove that in the professor's face.

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#37

Post by mgkimsal » Sun, Jul 06, 2014, 7:43 PM UTC

lostlogin wrote: post #9

ZoFreX wrote: post #7

Earlier quoted context omitted.

Time to get in touch with the press. 25 bugs in something health-critical? They'll have a field day.

My hope (which is steadily fading) was to make contact with someone in the development team to suggest a few improvements and show a few of the more obscure bugs that I haven't reported. I'd love to be able to positively influence the development. There has been limited "end user" input as far as I can tell and some small changes would make the platform so much more powerful. The upside to the lack of communication i…

They're waiting for patents on all the stuff you suggested before contacting you back to say that you've violated their intellectual property. Don't worry. You'll hear from them soon enough...

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#38

Post by vardump » Sun, Jul 06, 2014, 9:16 PM UTC

lostlogin wrote: post #5

>>As a result of the Therac-25 accidents, the FDA now requires documentation on software for new medical and other products: a paper trail, in other words, that can be examined by an independent body and retraced for flaws. Anyone have any idea if this can be looked at by the end user? I'm not a radiation technologist of the flavour mentioned in the article, I'm on the diagnostic side. I use an MR scanner with numero…

Like others have told, any complaint you make must be processed by the company. If it endangers human safety, failure to do so is against FDA's rules and can have dire and expensive consequences to the company in question.

Fixing bugs in a medical device is not slow because it'd take a long time to fix the bug itself. It's slow because there's a lot of paperwork (Device History Files, specifications, tons of reviews, etc.) and system testing that follows. That's why you'd only correct issues that do not endanger patient safety once there are enough or when they can be combined with new features in firmware.

There's also always a risk that those fixed issues cause patient endangering bugs.

It can take almost one year from implementing the fix until it is running on any significant number of medical (imaging) devices in question.

FDA's rules are there for a reason. Although... someone should tell FDA git exists. It's silly to revision things manually in Word documents.

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#39

Post by vardump » Sun, Jul 06, 2014, 9:23 PM UTC

kohanz wrote: post #22

lostlogin wrote: post #14

Earlier quoted context omitted.

Thanks - this has been in the back of my mind and is a reason I'm trying to be patient. A 2 line message saying what was happening would remove my frustration. Usually I get a corporate speak reply with a suggestion it is my fault though. What does the FDA code review do? If it isn't catching bugs that take the scanner offline for hours at a time, what is the point?

I've worked on several FDA-regulated products and have never had the FDA review my code. I would guess this only happens in extenuating circumstances. The FDA does not have the resources to do this for most products out there. We are required, however, to review our own code and maintain records of those reviews.

Exactly. FDA doesn't review code!

If there are complaints, FDA does sometimes review is the mountain of device related documentation. Design, assembly, maintenance, end user manuals, etc. Checking the paper trail. Is the paperwork done correctly, signed by a competent employee and reviewed by appropriate persons. There also needs to be watertight trail of employee training. Failure to have that does not end well!

Traceability (both physical and code) is another thing you better get right as a medical company. You need to know where, when, etc. each major component of the device came to be.

Medical companies literally generate so much paperwork, that separate storage facilities are needed for it. While you'd obviously have it in digital format for yourself, all of it is also printed out and signed.

Re: Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

#40

Post by vardump » Sun, Jul 06, 2014, 9:35 PM UTC

lostlogin wrote: post #9

ZoFreX wrote: post #7

Earlier quoted context omitted.

Time to get in touch with the press. 25 bugs in something health-critical? They'll have a field day.

My hope (which is steadily fading) was to make contact with someone in the development team to suggest a few improvements and show a few of the more obscure bugs that I haven't reported. I'd love to be able to positively influence the development. There has been limited "end user" input as far as I can tell and some small changes would make the platform so much more powerful. The upside to the lack of communication i…

There are so many layers between you and the developer... Without going to specifics, probably about 5. There's pretty much no chance you get to interact with the developers directly.