It's true that writing software requires only computers, and that's it would be too expensive to test it in real situations, when the stakes are high maybe it's also important to do live testing ?
How Is Critical Life or Death Software Tested?
11–20 of 161 posts
Re: How Is Critical Life or Death Software Tested?
#12It depends on the software budget and managers. When I worked a company one the CEO's had an idea about adding a camera to road semaphores so that cars wouldn't stop is the road is empty. I asked: What happens if the sun light hits the camera too much? The guy laughed in my face and told I was being ridiculous. I left the company some months later for other reasons but It was pretty scary to me to hear such words at that time. If you're writing software that might cost someone else's life, the budget shouldn't be a limitation nor time. I'm not aware of any software security regulation, If I'm going to release critical software, Does some entity exists to validates my code? This kind of stuff should exist to block companies that only see the profit of getting a contract.
Re: How Is Critical Life or Death Software Tested?
#13The article bit misleading, almost suggests that software correctness is achieved by testing, while this is definitely not true. Just for the record, none of the other technical fields using testing for safety critical engineering products as the main solution to ensure safety. This is not too different for software components in mission critical use cases. There are several ways to build a reliable system out of unreliable parts (like combining 3 different units by the 2 out of 3 principle, etc.) but testing is just not the way. It is a nice to have thing.
Re: How Is Critical Life or Death Software Tested?
#14For people interested in the idea of formal verification, you may want to look at TLA+ (https://en.wikipedia.org/wiki/TLA%2B) and PlusCal (https://en.wikipedia.org/wiki/PlusCal), which have been mentioned on HN before.
They're a specialty system for writing code (and mathematical proofs) where every possible system behavior for a given range of inputs can be examined for safety (outputs within allowed ranges with no unexpected behavior) and for liveliness (the expected progression from one output to another).
Re: How Is Critical Life or Death Software Tested?
#15I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…
> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running.
I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are performed. At most, there's some sort of metrics and server monitoring, but nothing in the actual software.
It's a work of fiction, but Star Trek has these "diagnostics" that are run, with several levels of correctness checks, for pretty much everything. In your typical app, it could be useful to ask it to "run diagnostics", to see if everything was still performing as expected.
Re: How Is Critical Life or Death Software Tested?
#16It's essentially an analog of the concept of tolerance in the physical world of manufacturing and assembly. The less your tolerance for error the more formal and carefully controlled the process, and the more money spent in testing, verification, feedback, and improvement. And yet you can still measure one value in the metric system and another in English units and drill a smoking hole in Mars. It was sort of strikin…
> statement about the software being bug free
I bet they required the guy who delivered the rocket fuel to sign something saying it contained no impurities, the guy who delivered the external tank to sign something saying it did not leak, etc... why should the software guy be special?
Yeah I know. We're special. But the world doesn't always see it that way.
Re: How Is Critical Life or Death Software Tested?
#17> As for the code itself, its perfection came as the result of basically the opposite of every trope normally assigned to "coder." Creativity in the shuttle group was discouraged; shifts were nine-to-five; code hotshots and superstars were not tolerated; over half of the team consisted of women; debugging barely existed because mistakes were the rarest of occurrences. Programming was the product not of coders and engineers, but of the Process.
Serious question: where do I get a job like this? It's my dream way of programming professionally.
Re: How Is Critical Life or Death Software Tested?
#18I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating th…
I've seen some software that does lots of tests on startup. One software I used (ATG) validated all the ORM mappings against the currently configured database on startup by default.
Re: How Is Critical Life or Death Software Tested?
#19I'm currently writing software that is non-critical for satellites. It's non-critical in the manner that if we get things wrong our company will lose millions of dollars but the satellite won't burn any resources that it can't get back.
We are currently porting code over from C++ to a C# system with parallel computation. The current system has been flying for a long time but has no testing and is tied to a bad UI. So we are re-writing.
That said, accuracy is number one. We have a pretty solid method for testing so far. We know have some robust input scenarios and we know that we want to get a specific output. So we are able to do fairly robust automated "regression" testing. If the numbers don't match then we have an issue, and we have to fix that before moving on.
After every validation that the new code gives us acceptable margins of error we wrap it up with unit tests so that we can then modify the code to try to optimize. Our testing is integrated from the highest level of the code down (I know backwards) but that's how we know we can validate the input.
We have a lot of testing and a long schedule. If this weren't critical software we'd have a much shorter turn around on what we are writing. We also work very closely with subject matter experts on every change we make. We have a guy who's been working with this software (and the subsequent theory) for 20 years. He's open to change, but he also validates everything so we don't accidentally change the output when we're optimizing.
Re: How Is Critical Life or Death Software Tested?
#20Not sure if true, but I heard a couple of researchers in my uni wrote some code for Boeing. They wrote the code in Prolog. I assume because it's easier to formal test it. And they didn't trust the compiler, so they had to check the generated machine code line by line. Apparently.