Live data from Hacker News

Software engineering lessons from RCAs of greatest disasters

anoopdixith.com

101–110 of 150 posts

Re: Software engineering lessons from RCAs of greatest disasters

#101
post #58

Earlier quoted context omitted.

So the article had a list of disasters and some useful lessons learned in its left and center columns. It also had lists of truisms about software engineering in the right column. They had nothing fundamental to do with each other. For instance, it tries to draw an equivalence between "Titanic's Captain Edward Smith had shown an "indifference to danger [that] was one of the direct and contributing causes of this unne…

So let's take those disasters and list the lessons that you would have learned from them. That's the way to constructively approach an article like this, out-of-hand dismissal is just dumb and unproductive. FWIW I've seen the leaders of software teams all the way up to the CTO run around like headless chickens during (often self inflicted) crisis. I think the biggest lesson from the Titanic is that you're never invul…

There are two useful lessons from the Titanic that can apply to software:

1) Marketing that you are super duper and special is meaningless if you've actually built something terrible (the Titanic was not even remotely as unsinkable as claimed, with "water tight" compartments that weren't actually watertight)

2) When people below you tell you "hey we are in danger", listen to them. Don't do things that are obviously dangerous and make zero effort to mitigate the danger. The danger of atlantic icebergs was well understood, and the Titanic was warned multiple times! Yet the captain still had inadequate monitoring, and did not slow down to give the ship more time to react to any threat.

Re: Software engineering lessons from RCAs of greatest disasters

#102
Instead of attempting to design a perfect system that cannot fail, the idea is to design a system that can tolerate failure of any component. (This is how airliners are designed, and is why they are so incredibly reliable.)

Safe Systems from Unreliable Parts https://www.digitalmars.com/articles/b39.html

Designing Safe Software Systems part 2 https://www.digitalmars.com/articles/b40.html

Re: Software engineering lessons from RCAs of greatest disasters

#103

Earlier quoted context omitted.

Lest someone think this is purely hypothetical: https://en.wikipedia.org/wiki/USS_Tang_(SS-306)

A mark 14 torpedo actually sinking something? What a bad stroke of luck!

The Mark 14 ended-up being a really good torpedo by the end of WWII.

It even remained in service until the 80ies.

In truth, and going back to this subject, the Mark 14 debacle highlights the need for a good and unbiased QA.

This also holds true for software engineering.

Re: Software engineering lessons from RCAs of greatest disasters

#104

One of my favourites that I ever heard about was from a friend of mine who used to work on safety-critical systems in defense applications. He told me fighter jets have a safety system that disables the weapons systems if a (weight) load is detected on the landing gear so that if the plane is on the ground and the pilot bumps the wrong button they don't accidentally blow up their own airbase[1]. So anyway when the Eu…

> Yes this means that scene from the cheesy action movie (can't remember which one) where Arnold Schwartzenegger finds himself on the ground in the cockpit of a russian plane and proceeds to blow up all the badguys while on the ground couldn't happen in real life. I think you meant "Tomorrow Never Dies" and the actor was Pierce Brosnan. Took me forever to find that, is it the right one?

Yeah maybe. I think that sort of rings a bell.

Re: Software engineering lessons from RCAs of greatest disasters

#105
post #59

Earlier quoted context omitted.

This makes no sense and is difficult to even respond to coherently. > It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, Either you're completely wrong, because we "as an industry" still push bugs and security flaws, or you're comparing two completely different things. > doing that on applications with a supe…

I would generally pass this comment by, but it's just so distastefully hostile because you totally missed the point. GP's comment was expressing sardonic disbelief that a modern jet wouldn't be able to receive remote software updates, considering it's so ubiquitous and reliable in other fields, even those with much, much lower costs. Not that developers don't release faults.

People tend to opine on systems engineering as if we had some sort of information superconductor connecting all minds involved.

Systems are Hard and complex systems are Harder. Thinking of entire class of failures as 'solved' is kinda like talking about curing cancer. There isn't one thing called cancer, there's hundreds.

There's no way to solve complex systems problems for good. Reality, technologies, tooling, people, language, everything changes all the time. And complex systems failure modes that happen today will happen forever.

Re: Software engineering lessons from RCAs of greatest disasters

#106
post #59

Earlier quoted context omitted.

This makes no sense and is difficult to even respond to coherently. > It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, Either you're completely wrong, because we "as an industry" still push bugs and security flaws, or you're comparing two completely different things. > doing that on applications with a supe…

I would generally pass this comment by, but it's just so distastefully hostile because you totally missed the point. GP's comment was expressing sardonic disbelief that a modern jet wouldn't be able to receive remote software updates, considering it's so ubiquitous and reliable in other fields, even those with much, much lower costs. Not that developers don't release faults.

Ahh, then I did misread it entirely. Thanks for stopping by to call me out.

It's still probably not a matter of capability... I wouldn't be so cavalier about software updates on my phone if it was holding me thousands of feet above the ground at the time.

Re: Software engineering lessons from RCAs of greatest disasters

#107
post #59

Earlier quoted context omitted.

This makes no sense and is difficult to even respond to coherently. > It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, Either you're completely wrong, because we "as an industry" still push bugs and security flaws, or you're comparing two completely different things. > doing that on applications with a supe…

I would generally pass this comment by, but it's just so distastefully hostile because you totally missed the point. GP's comment was expressing sardonic disbelief that a modern jet wouldn't be able to receive remote software updates, considering it's so ubiquitous and reliable in other fields, even those with much, much lower costs. Not that developers don't release faults.

Remote software updates on military vehicles? Hasn't anyone seen the new Battlestar Galactica? :)

Re: Software engineering lessons from RCAs of greatest disasters

#108
post #18
post #6

Another train crash that holds a valuable lesson was https://en.wikipedia.org/wiki/Eschede_train_disaster This demonstrates that sometimes "if you see something, say something" isn't enough - if a large piece of metal penetrates into the passenger compartment of a train from underneath, it's better to take the initiative and pull the emergency brake yourself .

Not entirely true: > Dittmann could not find an emergency brake in the corridor and had not noticed that there was an emergency brake handle in his own compartment. The learning from that should maybe instead be to keep non-technical management out of engineering decisions. The Wikipedia article fails to mention there was a specific manager who pushed the new wheel design into production and then went on to have a lo…

I dislike the current trend of calling lessons "learnings". I don't understand the shift in meaning. Learning is the act of acquiring knowledge. The bit of knowledge acquired has a long established name: lesson. What's the issue with that?

Re: Software engineering lessons from RCAs of greatest disasters

#109

Earlier quoted context omitted.

> Yes this means that scene from the cheesy action movie (can't remember which one) where Arnold Schwartzenegger finds himself on the ground in the cockpit of a russian plane and proceeds to blow up all the badguys while on the ground couldn't happen in real life. I think you meant "Tomorrow Never Dies" and the actor was Pierce Brosnan. Took me forever to find that, is it the right one?

Yeah maybe. I think that sort of rings a bell.

https://youtu.be/hcIgZ4kJ5Ow?t=340

this?

Re: Software engineering lessons from RCAs of greatest disasters

#110

Earlier quoted context omitted.

So let's take those disasters and list the lessons that you would have learned from them. That's the way to constructively approach an article like this, out-of-hand dismissal is just dumb and unproductive. FWIW I've seen the leaders of software teams all the way up to the CTO run around like headless chickens during (often self inflicted) crisis. I think the biggest lesson from the Titanic is that you're never invul…

There are two useful lessons from the Titanic that can apply to software: 1) Marketing that you are super duper and special is meaningless if you've actually built something terrible (the Titanic was not even remotely as unsinkable as claimed, with "water tight" compartments that weren't actually watertight) 2) When people below you tell you "hey we are in danger", listen to them. Don't do things that are obviously d…

Good stuff, thank you. This is useful, and it (2) ties into the Challenger disaster as well.
Post reply on HN