How Is Critical Life or Death Software Tested? • Hacker News Board

Re: How Is Critical Life or Death Software Tested?

#141

Post by henrik_w » Tue, Jun 02, 2015, 7:26 AM UTC

The story of the Boeing engineers flying on the test flights is a perfect example of "skin in the game" (from Antifragile by Nassim Nicholas Taleb).

Here's what I wrote about that in a blog post on Antifragility and SW development:

At the end of the book, there is a chapter on ethics that Taleb calls “skin in the game”. To have skin in the game, you should share both in the upside and downside. Taleb quotes the 3,800 year old Hammurabi’s code: “If a builder builds a house and the house collapses and causes the death of the owner – the builder shall be put to death”. It is interesting to view this from a software development perspective. I have never worked on software where people’s lives were in danger if the software failed, but I would not be willing to submit to Hammurabi’s code if I did. But I think a little less extreme form of skin in the game is actually very good. Being on call for example. If the software you wrote fails, you may get called in the middle of the night to help fix it. I have been on call at most of the places I have worked in the past, and I think it has a lot of benefits. It gives you an incentive to be very thorough, both in development and testing. It also forces you to make the software debuggable – otherwise you yourself will suffer. Another way of introducing skin in the game is dog-fooding – using the software you are developing in your daily work. I have never worked on software that we have been able to dog-food, but I think that is another great practice.

http://henrikwarne.com/2014/06/08/antifragility-and-software...

Re: How Is Critical Life or Death Software Tested?

#142

Post by peteretep » Tue, Jun 02, 2015, 7:43 AM UTC

z3t4 wrote: post #137

peteretep wrote: post #123

Earlier quoted context omitted.

If an error is caught and handled, calling it a crash seems disingenuous.

One mistake that ppl do is they wrap their code around a try ... catch, where it's better to throw an error and exit. If there's an error in one place, chances are there are also errors elsewhere, so it's better to restart the program instead of continue with a bad state. When the error gets thrown in your face, there's a higher chance that it gets fixed. But this also have its setbacks. Loosing the whole state can b…

I'm really note sure why you think catching an error in a separate process is somehow superior to catching it in a higher scope

Re: How Is Critical Life or Death Software Tested?

#143

Post by mbrameld » Tue, Jun 02, 2015, 7:51 AM UTC

nchelluri wrote: post #102

airza wrote: post #32

My experience with testing avionics controllers was: Everyone seemed to have the correct idea (that bugs were basically Not Allowed.) The company set up enough testing so that those bugs were eventually eliminated. However, the difference between good projects and bad projects was mostly the amount of time and money that this took. My favorite was one where we had an entire test harness written in Python that could c…

Care to elaborate on the potato chip bag story? Sounds intriguing.

My wild guess loosely based on my time spent writing software to analyze observed radar signals: I believe the parent probably opened the bag along the seams to get one flat piece of reflective (on the inside of the bag) material. I know weather balloons are often reflective so they can be tracked by radar. I'm guessing a full-size potato chip bag opened flat would be big enough for the radar to see, couldn't tell you if that's what the parent meant by fooling the radar or if something else was done to the bag to make the radar interpret it as an aircraft.

Re: How Is Critical Life or Death Software Tested?

#144

Post by qrybam » Tue, Jun 02, 2015, 7:54 AM UTC

aswanson wrote: post #103

noir_lord wrote: post #69

Earlier quoted context omitted.

> The parts I do have experience with it is utterly a miracle that anything works. Is my feeling about everything industry I've worked in. The stuff that runs telecoms (mostly billing side) particularly is the stuff of nightmares.

Seriously? Billing code? I would have imagined that code would be so subject to customer complaint that it would be forced into quality.

Can add anecdotal evidence that people are afraid of changing a company's billing system as it may result in bills being generated that are significantly different as to either raise client or internal concerns, so best to leave it alone, right?

Re: How Is Critical Life or Death Software Tested?

#145

Post by michaelt » Tue, Jun 02, 2015, 9:26 AM UTC

infinotize wrote: post #101

neurotech1 wrote: post #98

Earlier quoted context omitted.

An F-22 had a FCS failure and didn't realize takeoff. The pilot didn't do a IBIT on the FCS as required, therefore wasn't aware that all 3 rate sensors latched up. The jet was uncontrollable once it left the ground, and the pilot ejected safely. [0] As I recall, they modified the sensors to avoid latch up (extra pullup resister) and updated the FCS software to provide a warning if all 3 sensors return zero output. Ev…

To be fair this seems to be more of a classic example of a documentation or training problem. Right from your link: > During the mishap sequence, the MP started engines, perfomled an IBIT, and had a fully functioning Flight Control System. Subsequently, the MP shut down engines to allow maintenance personnel to service the Stored Energy System. During engine shut down, the MA's Auxiliary Power System (APU) was runnin…

  To be fair this seems to be more of a classic 
  example of a documentation or training problem.

Maybe - but if you were designing a consumer product, you wouldn't rely on the user following a checklist; the IBIT would run automatically when they turned on the ignition and sound an alarm (or even prevent takeoff) if the vehicle would be uncontrollable.

So you could also classify this as a user interface / design problem.

Re: How Is Critical Life or Death Software Tested?

#146

Post by gtirloni » Tue, Jun 02, 2015, 10:13 AM UTC

aswanson wrote: post #103

noir_lord wrote: post #69

Earlier quoted context omitted.

> The parts I do have experience with it is utterly a miracle that anything works. Is my feeling about everything industry I've worked in. The stuff that runs telecoms (mostly billing side) particularly is the stuff of nightmares.

Seriously? Billing code? I would have imagined that code would be so subject to customer complaint that it would be forced into quality.

I've worked in a support team for a telecom billing system and I was tasked with interacting with our development team to investigate bugs in production (and eventually moved to the development team). These systems are created just like any other commercial system, without any formal proofs and minimum requirement docs. To make things worse, they have to be flexible enough to support all and any billing plans that the business might come up with, so there is a lot of moving parts.

As other people have said here, nobody wants to touch it. Developers would often limit themselves to fix just a small portion of the code even though they thought the overall system could be improved in many ways, for fear of breaking something, causing a few million dollars of damage and getting fired. There was no assurance that any part of the systems should work like this or that.. only some vague expectations.

You're right, that would be a systems that should be built from scratch with that kind of concern but unfortunately it's not.

Re: How Is Critical Life or Death Software Tested?

#147

Post by grayarea » Tue, Jun 02, 2015, 10:39 AM UTC

outworlder wrote: post #15

acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are pe…

I used to write fire control and fire monitoring software. The whole idea of live testing is built into the ethos of such systems. In fire alarms and control the only tests that matter are the ones performed in a production environment.

Re: How Is Critical Life or Death Software Tested?

#148

Post by Tloewald » Tue, Jun 02, 2015, 11:30 AM UTC

clearf wrote: post #107

Tloewald wrote: post #30

Earlier quoted context omitted.

The project I'm working on (for the Federal government no less, and very much not life and death) involves having test data in the production system and tests that run in the production system. (I was impressed by this when I found out.)

Make sure they're well separated! c.f. http://www.wsj.com/articles/SB100014240527487033765045754919...

The stuff we are doing is so not life and death that nothing like this could happen. E.g. We have fairly insane levels of security for information that is inherently public.

Re: How Is Critical Life or Death Software Tested?

#149

Post by VLM » Tue, Jun 02, 2015, 12:41 PM UTC

fredmorcos wrote: post #62

acomjean wrote: post #8

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…

> Reliability was favored over other things (for example recursion was discouraged). This sounds really strange to me. So may I ask why? I find that recursion - most of the time - helps shorten and clarify the code. Also, doesn't recursion make induction proofs trivial?

Non recursion is waaay "easier" to troubleshoot finite precision floating point issues and failures of the first part of "be liberal in what you accept and conservative in what you send".

If you assume infinite precision arithmetic and a very friendly environment for inputs, recursion always looks simpler, but by the time you clean it up to handle real world issues, non recursion instead looks simpler.

Its too easy to write recursive end conditions along the lines of "if x == 42" when your helpful floating point routine somehow mysteriously rounded x to 42.00000001 so it'll never equal, or "no (supposedly) UTF-16 encoded string would ever have an odd number of bytes, even though I have no control of the source and the source is known to occasionally be insane" or at least thats how I remember it. I've run into both. Its not funny at the time but in retrospect its usually fairly hilarious.

Personally I think its harder for people to understand concurrency issues WRT recursion, but I'll probably just get flamed for that one. I feel more people have "leveled up" with concurrency and non-recursive code and functional style programming than have leveled up to include recursion in that mix. Imagine two (three?) concurrent recursive algos fighting each other over one data structure.

Re: How Is Critical Life or Death Software Tested?

#150

Post by jerf » Tue, Jun 02, 2015, 1:20 PM UTC

manarth wrote: post #39

jerf wrote: post #23

Earlier quoted context omitted.

This is why this discussion sometimes frustrates me. A lot of the defects we have are because you aren't willing to pay for the sort of software that wouldn't have defects. It's natural to read that as a sort of cynical accusation, but instead, I mean it straight... you really aren't willing to pay what it would take, and you shouldn't be. A $1000 Facebook-access app for your phone (that still somehow has some sort o…

"What do you mean you can't be bothered to run the automated test suite that I have handed you on a silver platter on your build server?" As devil's advocate, why not just run this for me (e.g. on every commit/every push)? Much like the web usability ethos "Why make me think" - why make me work? The lower the barrier to testing - ideally zero, it just happens without the dev having to do anything - the more testing w…

"As devil's advocate, why not just run this for me (e.g. on every commit/every push)?"

In my real-life experience with a multiple-team environment, which is where the question came from, my running the tests doesn't do any good if you're going to consider it "my" server and simply ignore the results.

The key point here isn't a technical one. The key point here was, as an professional engineer, how can you justify not taking such a great bang/buck quality option that will far, far more than pay for itself? Explaining how I can be a professional engineer on your behalf more than misses the point.