Software engineering lessons from RCAs of greatest disasters

> Every disaster is preventable.

No, there is such a thing as residual risk and there are always disasters that you can't prevent such as natural disasters. But even then you can have risk mitigation and strategies for dealing with the aftermath of an incident to limit the effects.

> Everything on the list was happening in human-engineered environments - as do most things that affect humans.

That is precisely why they were picked and make for good examples.

> The human race has been the master of its own destiny since the 1900s.

That isn't true and it likely will never be true. We are tied 1:1 to the fate of our star and may well go down with it. There is a small but non-zero chance that we can change our destiny but I wouldn't bet on it. And even then in the even longer term it still won't matter. We are passengers, the best we can do is be good stewards of the ship we've inherited.

> The questions are how far before the disaster we need to look to find somewhere to act and what needed to be given up to change the flow of events.

Indeed. So in the case of each of the items listed the RCA gives a point in time where the accident given the situation as it existed was no longer a theoretical possibility but an event in progress. Situation and responses determined how far it got and in each of the cases outlined you can come up with a whole slew of ways in which the risk could have been reduced and possibly how the whole thing may have been averted once the root cause had triggered. But that doesn't mean that the root cause doesn't matter, it matters a lot. But the root cause isn't always a major thing. An O-ring, a horseshoe...

> But that doesn't have any implications for software engineering.

If that is your takeaway then for you it indeed probably does not. But I see such things in software engineering every other week or so and I think there are many lessons from these events that apply to software engineering. As do the people that design reliable systems, which is why many of us are arguing for liability for software. Because once producers of software are held liable for their product a large number of the bad practices and avoidable incidents (not just security) would become subject to the Darwinian selection process: bad producers would go out of business.

> Studying a software failure post mortem will be a lot more useful than studying 9/11.

You can learn lots of things from other fields, if you are open to learning in general. Myopically focusing on your own field is useful and can get you places but it will always result in 'deep' approaches, never in 'wide' approaches and for a really important system both of these approaches are valid and complementary.

To make your life easier the author has listed in the right hand column which items from the non-software disasters carry over into the software world, which I think is a valuable service. A middlebrow dismissal of that effort is throwing away an opportunity to learn, for free, from incidents that have all made the history books. And if you don't learn from your own and others' mistakes then you are bound to repeat that history.

Software isn't special in this sense. Not at all. What is special is the arrogance of some software people who believe that their field is so special that they can ignore the lessons from the world around them. And as a corollary: that they can ignore all the lessons already learned in software systems in the past. We are in an eternal cycle of repeating past mistakes with newer and shinier tools and we urgently need to break out of it.

Care to elaborate? I know professional engineers in Canada get a designation but I’m not aware of anything similar for software engineers.

Up to a point, but the "sometimes" makes it difficult to say anything definite. There's no shortage of stories where immediate intervention has made things worse, such as a burning train being stopped in a tunnel.

Furthermore, this sort of counterfactual or "if only" analysis can be used to direct attention away from what matters, as was done in the hounding of the SS Californian's captain during the inquiry into the sinking of the Titanic.

Here, one cannot fault the passenger for first getting himself and his family out of the compartment, and he correctly determined that the train manager's "follow the rules" response was inadequate in the circumstances - in fact, the inquiry might have considered the incongruity of having an emergency brake available for any passenger to use at any time, while restricting its use by train crew.

RCA quite properly focuses on the causes of the event, which would have been of equal significance even if the train had been halted in time, and which would continue to present objective risks unless addressed.

One of my favourites that I ever heard about was from a friend of mine who used to work on safety-critical systems in defense applications. He told me fighter jets have a safety system that disables the weapons systems if a (weight) load is detected on the landing gear so that if the plane is on the ground and the pilot bumps the wrong button they don't accidentally blow up their own airbase[1]. So anyway when the Eurofighter Typhoon did its first live weapons test the weapons failed to launch. When they did the RCA they found something like[2]

    bool check_the_landing_gear_before_shootyshoot(double weight_from_sensor, double threshold) {
        //FIXME: Remember to implement this before we go live
        return false
    }

So when the pilot pressed the button the function disabled the weapons as if the plane had been on the ground. Because the "correctness" checks were against the Z spec and this function didn't have a unit test because it was deemed too trivial, the problem wasn't found before launch, so this cost several millions to redeploy the (one-line) fix to actually check the weight from the sensor was less than the threshold.

[1] Yes this means that scene from the cheesy action movie (can't remember which one) where Arnold Schwartzenegger finds himself on the ground in the cockpit of a russian plane and proceeds to blow up all the badguys while on the ground couldn't happen in real life.

[2] Not the actual code which was in some weird version of ADA apparently.

Close call. I know of a similar case where a fire suppression systems check ended up with massive data loss in a very large storage array.