Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is Critical Life or Death Software Tested?

motherboard.vice.com

Post reply on HN
Print view View original

51–60 of 161 posts

Re: How Is Critical Life or Death Software Tested?

#51

Post by david-given »

Aleman360 wrote: post #45
manarth wrote: post #43

Earlier quoted context omitted.

You just have to look at Therac 25 for the risks of relying on software interlocks alone. One of the prevailing pieces of feedback was the lack of hardware interlocks - whether that's possible on anesthesia machines I don't know…but the prevailing wisdom is to use/include hardware interlocks wherever that's feasible, for any critical life-supporting equipment.

Not sure what you mean by "interlocks", but the hardware was quite distributed. Each critical component had its own board and industrial microcontroller. And we had various levels of watchdogs keeping track of system health at all times.

Interlocks are usually fairly crude safety measures, normally in hardware, to make sure that particular combinations of events cannot happen.

The Therac-25 is a famous comp.risks cautionary tale. Among the many, many design misfeatures (if you haven't come across it, it's worth a read) was the one that killed people:

It was capable of providing two kinds of radiation therapy; electron beam radiation and X-ray radiation. It worked by having an electron beam generator which could be operated at either high power or lower power. Low power was used directly. High power was only used to irradiate a tungsten target which produced X-rays. (I'm simplifying here.)

You can probably guess what went wrong; people were directly exposed to the high power electron beam. Several of them died.

The obvious interlock here (which apparently previous versions had) was to have a mechanical switch which would only enable the high-power beam when the tungten target was rotated into place. No target, no high power. Simple and relatively foolproof (although it's possible for interlocks to go wrong too).

Re: How Is Critical Life or Death Software Tested?

#52

Post by david-given »

Aleman360 wrote: post #47
click170 wrote: post #44

Earlier quoted context omitted.

How does an anesthesia machine fail safe? What I mean is, does it continue applying anesthetic in the event of a power failure or does that stop entirely?

It can operate completely mechanically. There is an integrated UPS in case mains fails. If the battery also fails, a pneumatic whistle goes off to alert the user. If for some reason something goes really bad, they switch to using an ambu bag, typically hung on the back of the machine.

I like the idea of the pneumatic whistle. A pressure vessel with a sprung loaded valve held closed by a solenoid, maybe? However it works, it's a really neat piece of lateral thinking.

Re: How Is Critical Life or Death Software Tested?

#54

Post by david-given »

bargl wrote: post #19

I'm currently writing software that is non-critical for satellites. It's non-critical in the manner that if we get things wrong our company will lose millions of dollars but the satellite won't burn any resources that it can't get back. We are currently porting code over from C++ to a C# system with parallel computation. The current system has been flying for a long time but has no testing and is tied to a bad UI. So…

C# is an odd choice for avionics software, surely? It's got a really heavyweight runtime and nondeterministic behaviour due to the garbage collector. I assume this is the satellite's application layer and isn't real time?

...and what are you running it on? I wasn't aware of any embedded operating systems which supported C#!

Re: How Is Critical Life or Death Software Tested?

#55

Post by tedunangst »

superuser2 wrote: post #33
eck wrote: post #16

Earlier quoted context omitted.

> statement about the software being bug free I bet they required the guy who delivered the rocket fuel to sign something saying it contained no impurities, the guy who delivered the external tank to sign something saying it did not leak, etc... why should the software guy be special? Yeah I know. We're special. But the world doesn't always see it that way.

No. That it contained under n ppm of impurities, that it did not leak more than 0.001%/year, maybe. Only a Sith (and consumer product marketing that is lying to you) deals in absolutes. In the durable goods world, you don't pretend things are perfect. Failure modes are designed and disclosed, replacement of parts is expected and made reasonable, tolerances are marked, failure rate metrics like MTBF are known, and as…

At least some consumer goods have similar limits, e.g. maximum amount of arsenic permitted in apple juice. Of course, whenever people find out about these limits, they lose their shit... "Why is any arsenic ok? OMG! The evil government is trying to poison us!"

Re: How Is Critical Life or Death Software Tested?

#56

Post by yitchelle »

cesarbs wrote: post #17

> As for the code itself, its perfection came as the result of basically the opposite of every trope normally assigned to "coder." Creativity in the shuttle group was discouraged; shifts were nine-to-five; code hotshots and superstars were not tolerated; over half of the team consisted of women; debugging barely existed because mistakes were the rarest of occurrences. Programming was the product not of coders and eng…

Think companies that build products that deals with human safety, ie automotive, military, medical, aerospace etc. Each of those industry will have their own definition of what is a good process is, and some are much stricter than others.

I work in automotive, so it is govern by processes such as ASPICE and ISO26262.

Re: How Is Critical Life or Death Software Tested?

#57

Post by lambdaelite »

cesarbs wrote: post #17

> As for the code itself, its perfection came as the result of basically the opposite of every trope normally assigned to "coder." Creativity in the shuttle group was discouraged; shifts were nine-to-five; code hotshots and superstars were not tolerated; over half of the team consisted of women; debugging barely existed because mistakes were the rarest of occurrences. Programming was the product not of coders and eng…

I'm my experience, engineering firms that work on software-intensive projects.

Re: How Is Critical Life or Death Software Tested?

#58

Post by digikata »

cesarbs wrote: post #17

> As for the code itself, its perfection came as the result of basically the opposite of every trope normally assigned to "coder." Creativity in the shuttle group was discouraged; shifts were nine-to-five; code hotshots and superstars were not tolerated; over half of the team consisted of women; debugging barely existed because mistakes were the rarest of occurrences. Programming was the product not of coders and eng…

Any field where the bug could be catastrophic. The only reason the shuttle group was sustainable the way it operated was because a bug in the software they worked upon was in that category. Off hand, commercial space, deep space science, aviation, medical, nuclear, & military systems (not all) are that way. Be warned that it's very slow moving, and your skills in relatively new hardware/software stacks will basically go out of date. Your skills will be in being a specialist in whatever the environment is of the system you're working upon - much more so than being a general software engineer. It's not bad, but it's very different than the typical HN environments.

Re: How Is Critical Life or Death Software Tested?

#59

Post by lambdaelite »

Aleman360 wrote: post #45
manarth wrote: post #43

Earlier quoted context omitted.

You just have to look at Therac 25 for the risks of relying on software interlocks alone. One of the prevailing pieces of feedback was the lack of hardware interlocks - whether that's possible on anesthesia machines I don't know…but the prevailing wisdom is to use/include hardware interlocks wherever that's feasible, for any critical life-supporting equipment.

Not sure what you mean by "interlocks", but the hardware was quite distributed. Each critical component had its own board and industrial microcontroller. And we had various levels of watchdogs keeping track of system health at all times.

A familiar interlock would be the mechanism used to disable a microwave magnetron when the door is in the open position.

Re: How Is Critical Life or Death Software Tested?

#60

Post by ploxiln »

manarth wrote: post #43
Aleman360 wrote: post #37

Worked on anesthesia machines for a few years. Since both hardware and software is involved, the testing was quite extensive. * Lots of manual testing. While we did unit testing and some automated integration testing, most defects were found using exhaustive manual testing by trained engineers. * Randomized UI testing. Used UI automation to exercise the UI with various physical configurations of the system. Would oft…

You just have to look at Therac 25 for the risks of relying on software interlocks alone. One of the prevailing pieces of feedback was the lack of hardware interlocks - whether that's possible on anesthesia machines I don't know…but the prevailing wisdom is to use/include hardware interlocks wherever that's feasible, for any critical life-supporting equipment.

I don't disagree that "hardware interlocks" are a good idea for such equipment. But now that I think about it, I'm annoyed that we can't know if software was competently written and tested - is it really that different from hardware in that respect?

How about, we require that the source be open, and if it's too convoluted for the hospital's respected experts to check, then it fails inspection?

Hardware uses many standard parts and materials, and similarly, the software could use a few plain-simple-standard libraries, like libc (but not the floating point functions), zlib, libpng.

The Therac 25 case was just incompetence. The vendor was told about the problem, but was in denial, then later supplied a hardware fix which didn't fix the problem. The problem had to be thoroughly investigated and proven by a doctor and operator over many months. Why couldn't the vendor have investigated more thoroughly themselves, in a week or so? Why weren't they more careful about race conditions? (The problem was triggered by a human able to type into the interface too fast. An actual human.)

Post reply on HN