How Is Critical Life or Death Software Tested? • Hacker News Board

Re: How Is Critical Life or Death Software Tested?

#91

Post by acranox » Mon, Jun 01, 2015, 11:47 PM UTC

mlonkibjuyhv wrote: post #61

acomjean wrote: post #26

Earlier quoted context omitted.

Even while running the radar would keep track of communications between the parts and make sure things were still ok. The system needed messages periodically from the external components and vice versa to make sure things were ok. There were status messages sent around too. And a display of how things were doing. Its been a while, but I remember some of the things. You could command the external things to run diagnos…

Cars often disable any system capable of interfering with the wheels at the least sign of issue. Insufficient seal on the fuel-cap? Disable ABS, TCS, etc.

That's not correct. Cars will generally have something that's often called a 'limp home' mode. I saw a car that's PCM (powertrain control module) that failed some internal test, so it fell back to a basic mode where the engine wouldn't rev over about 2000RPM, and I'm sure all the emissions systems went into a basic fail-safe mode, where things like the fuel delivery goes into a hard-coded mode, instead of using feedback from the oxygen sensors to tune the fuel delivery. But a loose fuel cap doesn't disable the ABS or TCS. Even a single faulty wheel speed sensor doesn't have to disable the entire ABS, it can still independently assess each wheel, determine if a lock-up is imminent and modulate the brakes for that wheel.

Re: How Is Critical Life or Death Software Tested?

#92

Post by snops » Mon, Jun 01, 2015, 11:48 PM UTC

david-given wrote: post #54

bargl wrote: post #19

I'm currently writing software that is non-critical for satellites. It's non-critical in the manner that if we get things wrong our company will lose millions of dollars but the satellite won't burn any resources that it can't get back. We are currently porting code over from C++ to a C# system with parallel computation. The current system has been flying for a long time but has no testing and is tied to a bad UI. So…

C# is an odd choice for avionics software, surely? It's got a really heavyweight runtime and nondeterministic behaviour due to the garbage collector. I assume this is the satellite's application layer and isn't real time? ...and what are you running it on? I wasn't aware of any embedded operating systems which supported C#!

I'm guessing its actually running on a ground station, providing commands or analysing data from satellites, hence why its non-critical and the need has arisen for parallel computation, presumably to speed it up.

Many ARM embedded systems can support C# with the open source .net micro framework (http://www.netmf.com), which doesnt require any OS, and was originally developed for Microsoft's SPOT watch. I haven't used it myself, and I agree with you that it doesn't sound, ideal at first for realtime apps, but none or soft realtime embedded applications are common too.

Re: How Is Critical Life or Death Software Tested?

#93

Post by robotresearcher » Mon, Jun 01, 2015, 11:49 PM UTC

fredmorcos wrote: post #62

acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Reliability was favored over other things (for example recursion was discouraged). This sounds really strange to me. So may I ask why? I find that recursion - most of the time - helps shorten and clarify the code. Also, doesn't recursion make induction proofs trivial?

tldr version: Because you want (i) an acyclic function call tree of (ii) predictable depth.

Re: How Is Critical Life or Death Software Tested?

#94

Post by Sophistifunk » Mon, Jun 01, 2015, 11:51 PM UTC

lambdaelite wrote: post #59

Aleman360 wrote: post #45

Earlier quoted context omitted.

Not sure what you mean by "interlocks", but the hardware was quite distributed. Each critical component had its own board and industrial microcontroller. And we had various levels of watchdogs keeping track of system health at all times.

A familiar interlock would be the mechanism used to disable a microwave magnetron when the door is in the open position.

Which apparently doesn't work so well, according to radio astronomers :-/

Kinda makes me worry about how many times I've microwaved the meat-n-two-veg just a bit.

Re: How Is Critical Life or Death Software Tested?

#95

Post by manarth » Mon, Jun 01, 2015, 11:51 PM UTC

pdkl95 wrote: post #87

david-given wrote: post #51

Earlier quoted context omitted.

Interlocks are usually fairly crude safety measures, normally in hardware, to make sure that particular combinations of events cannot happen. The Therac-25 is a famous comp.risks cautionary tale. Among the many, many design misfeatures (if you haven't come across it, it's worth a read) was the one that killed people: It was capable of providing two kinds of radiation therapy; electron beam radiation and X-ray radiati…

Really, every engineer should read the report[1] from the Therac-25 investigation. I would hope anybody that is working on anything that could be potentially dangerous has already read it. There problems in the Therac-25 went a lot further than just the bad design of the target-selection, which had an (badly designed) interlock. It checked that the rotating beam target was in the correct position to match the high/lo…

"The really telling thing, though, is how the previous model acted…it would blow fuses regularly"

Good. When a fuse blows, it shows something is wrong, and needs fixing. Replacing the fuse with a nail or something else that doesn't blow is a sure-fire way to set the thing on fire. Bad enough for a desk-lamp, a little worse for radiotherapy machine.

Sounds like people were irritated by fuses blowing, and decided to simply short-circuit the fuses instead.

Re: How Is Critical Life or Death Software Tested?

#96

Post by damoncali » Tue, Jun 02, 2015, 12:00 AM UTC

mlonkibjuyhv wrote: post #61

acomjean wrote: post #26

Earlier quoted context omitted.

Even while running the radar would keep track of communications between the parts and make sure things were still ok. The system needed messages periodically from the external components and vice versa to make sure things were ok. There were status messages sent around too. And a display of how things were doing. Its been a while, but I remember some of the things. You could command the external things to run diagnos…

Cars often disable any system capable of interfering with the wheels at the least sign of issue. Insufficient seal on the fuel-cap? Disable ABS, TCS, etc.

Funny. Just the other day, my gas cap was loose. The traction control light came on along with the check engine light. As far as I could tell, it (traction control) was still operational, however.

Seemed like a funny combination of lights to blink on.

Re: How Is Critical Life or Death Software Tested?

#97

Post by mitchty » Tue, Jun 02, 2015, 12:39 AM UTC

manarth wrote: post #86

[deleted] wrote: post #80

[deleted]

"ship both a and b, whose output always has to match exactly or it reports a fault and the component requires replacement." And when that software disagrees…but the plane happens to be at 40,000 feet? The plane just stops running until the component is replaced? I don't know too much about plane hardware, but I scuba-dive a rebreather which has critical life-support electronics. It has two independent computers, and…

I have a friend that used to do embedded programming for nuclear power plants.

From what he said they have 5 systems, two running software from vendor a, two from vendor b, and another running software from vendor c for failsafe. A quorum have to agree, if any don't it gets pulled out of service and analyzed why it wasn't the same output. If you get down to one there is still a mechanical failover but at that point you're already bringing things down. 3 at any time was considered a critical failure. Two at once would be full stop time.

I'd presume planes are similar. I know the space shuttle did the same at least so I assume the technique is common.

Re: How Is Critical Life or Death Software Tested?

#98

Post by neurotech1 » Tue, Jun 02, 2015, 12:47 AM UTC

dugmartin wrote: post #64

outworlder wrote: post #15

Earlier quoted context omitted.

> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are pe…

Early in my career I worked on military flight data recorders, including the development of the software for the F-22's "black box". Those systems have SBIT, IBIT, PBIT and MBIT sub-systems were BIT is "built in test" and S = startup, I = initiated, P = periodic and M = maintenance. I remember making the Star Trek diagnostic joke myself when I was assigned the SBIT work. Each BIT does varying level of testing based o…

An F-22 had a FCS failure and didn't realize takeoff. The pilot didn't do a IBIT on the FCS as required, therefore wasn't aware that all 3 rate sensors latched up. The jet was uncontrollable once it left the ground, and the pilot ejected safely. [0]

As I recall, they modified the sensors to avoid latch up (extra pullup resister) and updated the FCS software to provide a warning if all 3 sensors return zero output. Even though this wasn't really an error in the FCS software, It could be argued that it failed to detect an erroneous input from 3 redundant sensors that latched up for the same reason.

This is a classic example of a pilot who misunderstood the pre-takeoff checklist procedure and cost the USAF a $325m aircraft.

[0] http://usaf.aib.law.af.mil/ExecSum2005/F-22A_20Dec04.pdf

Re: How Is Critical Life or Death Software Tested?

#99

Post by albinofrenchy » Tue, Jun 02, 2015, 12:51 AM UTC

fredmorcos wrote: post #62

acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Reliability was favored over other things (for example recursion was discouraged). This sounds really strange to me. So may I ask why? I find that recursion - most of the time - helps shorten and clarify the code. Also, doesn't recursion make induction proofs trivial?

In addition to what others have said here; often recursive calls can also trivially be optimized into loops by the compiler.

This is very convenient in most cases, but could hide the fact that a direct recursive call will overflow your stack.

This isn't apparent and would test out ok until someone makes a small; seemingly insignificant change which the compiler can't do tail call optimization on, and all of a sudden things fail.

Re: How Is Critical Life or Death Software Tested?

#100

Post by derekp7 » Tue, Jun 02, 2015, 12:56 AM UTC

dugmartin wrote: post #64

outworlder wrote: post #15

Earlier quoted context omitted.

> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are pe…

Early in my career I worked on military flight data recorders, including the development of the software for the F-22's "black box". Those systems have SBIT, IBIT, PBIT and MBIT sub-systems were BIT is "built in test" and S = startup, I = initiated, P = periodic and M = maintenance. I remember making the Star Trek diagnostic joke myself when I was assigned the SBIT work. Each BIT does varying level of testing based o…

What types of errors would cause one of the tests to fail? Is it mostly testing for hardware errors, or are there any software logic errors that could make it to production, but be caught by one of the tests several months down the road? The only software related items I can think of are edge cases where a built in test is based on real time input. Kind of like running the calculations through multiple independent implementations of the software.