Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is Critical Life or Death Software Tested?

motherboard.vice.com

Post reply on HN
Print view View original

61–70 of 161 posts

Re: How Is Critical Life or Death Software Tested?

#61

Post by mlonkibjuyhv »

acomjean wrote: post #26
outworlder wrote: post #15

Earlier quoted context omitted.

> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are pe…

Even while running the radar would keep track of communications between the parts and make sure things were still ok. The system needed messages periodically from the external components and vice versa to make sure things were ok. There were status messages sent around too. And a display of how things were doing. Its been a while, but I remember some of the things. You could command the external things to run diagnos…

Cars often disable any system capable of interfering with the wheels at the least sign of issue. Insufficient seal on the fuel-cap? Disable ABS, TCS, etc.

Re: How Is Critical Life or Death Software Tested?

#62

Post by fredmorcos »

acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Reliability was favored over other things (for example recursion was discouraged).

This sounds really strange to me. So may I ask why? I find that recursion - most of the time - helps shorten and clarify the code. Also, doesn't recursion make induction proofs trivial?

Re: How Is Critical Life or Death Software Tested?

#63

Post by phleet »

fredmorcos wrote: post #62
acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Reliability was favored over other things (for example recursion was discouraged). This sounds really strange to me. So may I ask why? I find that recursion - most of the time - helps shorten and clarify the code. Also, doesn't recursion make induction proofs trivial?

It's a lot harder to reason about memory constraints on recursive programs.

The clearness and correctness of the code often ignores the possibility for stack overflow. Most naive implementations of DFS will hit the stack limit given trees that are all one long path from a single root to a single leaf.

Re: How Is Critical Life or Death Software Tested?

#64

Post by dugmartin »

outworlder wrote: post #15
acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are pe…

Early in my career I worked on military flight data recorders, including the development of the software for the F-22's "black box". Those systems have SBIT, IBIT, PBIT and MBIT sub-systems were BIT is "built in test" and S = startup, I = initiated, P = periodic and M = maintenance. I remember making the Star Trek diagnostic joke myself when I was assigned the SBIT work.

Each BIT does varying level of testing based on it's runtime budget but there are a lot of very basic tests that don't make much sense until you see your first field report of some register bit "sticking". Its much better to ground a plane that can't add 1+1 than to find that out in mid-flight.

Re: How Is Critical Life or Death Software Tested?

#65

Post by pwnna »

markbnj wrote: post #29
pwnna wrote: post #4

Earlier quoted context omitted.

> It was sort of striking to read Charles Fishman's statement about the software being bug free, followed immediately by the supporting fact that the last three versions had one bug each. If they had one bug, how are you 100% sure they didn't have two? I think you're being overcritical here. It's really really really difficult to reach 100% in any real form of measurement. So I think when they mean "bug free", they p…

>> So I think when they mean "bug free", they probably mean the chance of a bug is below some threshold of probability. The famous six sigma rule comes in mind. No doubt, and that was in fact my point. I think most of us would be very reluctant to use the phrase "bug free," and in this case his statement was obviously meant to be in stark contrast to that reluctance.

I think this is more of a technicality that's very subjective. I feel that "no" depends on the probability of something happening as well as how catastrophic it is.

For example, after buying a lottery ticket, I can pretty comfortably say that I'm not going to win. Sure, there is a chance, but is it really going to happen? No.

As another example, if I put down a waterbottle on my bike seat, there's a 1% chance of the water spilling onto the sidewalk. At this point, I'm also comfortable saying that it's not going to happen. However, if the waterbottle has a 1% chance of exploding when I put it down, I don't think I'll be comfortable saying that it won't happen. The chance of that must be much much lower before I can accept it.

I would like to give the person who said that the benefit of the doubt. I'm sure they understand the implication of bug free, but it's likely just easier to say that rather than explaining stats/tradeoffs to a journalist.

Re: How Is Critical Life or Death Software Tested?

#66

Post by lambdaelite »

tedunangst wrote: post #55
superuser2 wrote: post #33

Earlier quoted context omitted.

No. That it contained under n ppm of impurities, that it did not leak more than 0.001%/year, maybe. Only a Sith (and consumer product marketing that is lying to you) deals in absolutes. In the durable goods world, you don't pretend things are perfect. Failure modes are designed and disclosed, replacement of parts is expected and made reasonable, tolerances are marked, failure rate metrics like MTBF are known, and as…

At least some consumer goods have similar limits, e.g. maximum amount of arsenic permitted in apple juice. Of course, whenever people find out about these limits, they lose their shit... "Why is any arsenic ok? OMG! The evil government is trying to poison us!"

The person in charge of a mass spectrometer (Thermo Element 2) at a local college likes to toy with his students by running tests on their commercial bottled water during a lab demonstration. This particular instrument can pick up minute traces of uranium, lead, and the like (on the order of fg/L IIRC).

At the end of the demonstration, the trash can is full of discarded water bottles.

Re: How Is Critical Life or Death Software Tested?

#68

Post by noir_lord »

fredmorcos wrote: post #62
acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Reliability was favored over other things (for example recursion was discouraged). This sounds really strange to me. So may I ask why? I find that recursion - most of the time - helps shorten and clarify the code. Also, doesn't recursion make induction proofs trivial?

The issue is largely that unbounded recursion is quite easy to do accidentally (in many of the languages that where used in the past) with the resulting stack smashing causing issues.

Also many of these systems where hard real time as in "if we don't respond in under 30ms something expensive goes bang" and again recursion can cause problems with that, lots of these systems are interrupt driven and have no garbage collection or threading so you can't just pre-empt them in that event since by the time you spot the problem you blew through your deadline and something went bang.

Re: How Is Critical Life or Death Software Tested?

#69

Post by noir_lord »

DannoHung wrote: post #48
acomjean wrote: post #46

Earlier quoted context omitted.

Large corporations with government contracts have this kind of work. Probably financial institutions too. The coding proces is slower than most people are used too and can become frustrating.

Financial institutions do not necessarily do things this way. Some parts might, but I don't have any experience with them. The parts I do have experience with it is utterly a miracle that anything works.

> The parts I do have experience with it is utterly a miracle that anything works.

Is my feeling about everything industry I've worked in.

The stuff that runs telecoms (mostly billing side) particularly is the stuff of nightmares.

Re: How Is Critical Life or Death Software Tested?

#70

Post by __z »

cesarbs wrote: post #17

> As for the code itself, its perfection came as the result of basically the opposite of every trope normally assigned to "coder." Creativity in the shuttle group was discouraged; shifts were nine-to-five; code hotshots and superstars were not tolerated; over half of the team consisted of women; debugging barely existed because mistakes were the rarest of occurrences. Programming was the product not of coders and eng…

Government contracting.
Post reply on HN