Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software engineering lessons from RCAs of greatest disasters

anoopdixith.com

Post reply on HN
Print view View original

61–70 of 150 posts

Re: Software engineering lessons from RCAs of greatest disasters

#61

Post by dacox »

tra3 wrote: post #34
gdevenyi wrote: post #26

This is why software engineering is a protected profession in some parts of the world (Canada at least), as civil responsibility and safety, along with formal legal liability is part of licensure

Care to elaborate? I know professional engineers in Canada get a designation but I’m not aware of anything similar for software engineers.

Engineer is a regulated term and profession in Canada, with professional designations like the P.eng - they get really mad when people the term engineer more loosely, as is common in the tech industry.

Because of this, there are "B.Seng" programs at some Canadian universities, as well as the standard "B.Sc" computer science program.

The degree was very new when I attended uni, so went for Comp sci intead as it seemed more "real". The B.Seng kids seemed to focus a lot more on industry things (classes on object oriented programming), which everyone picked up when doing internships anyways. They also had virtually no room for electives, whereas the CS calendar was stacked with very interesting electives which imo were vastly more useful in my career.

In practice, no one gives a hoot which degree you have, and we tend to just use the term SWeng regardless.

It honestly kinda feels like a bunch of crotchety old civil engineers trying to regulate an industry they're not a part of. I have _never_ seen a job require this degree.

Re: Software engineering lessons from RCAs of greatest disasters

#62

Post by jacquesm »

yuliyp wrote: post #58
jacquesm wrote: post #52

Earlier quoted context omitted.

> It is 2023. So? Mistakes are still being made, every day. Nothing has changed since the stone age except for our ability - and hopefully willingness - to learn from previous mistakes. If we want to. > The damage of natural disasters can be mitigated. You wish. > When the San Andreas fault goes it'll probably get an entry on that list with a "why did we build so much infrastructure on this thing? Why didn't we prepa…

So the article had a list of disasters and some useful lessons learned in its left and center columns. It also had lists of truisms about software engineering in the right column. They had nothing fundamental to do with each other. For instance, it tries to draw an equivalence between "Titanic's Captain Edward Smith had shown an "indifference to danger [that] was one of the direct and contributing causes of this unne…

So let's take those disasters and list the lessons that you would have learned from them. That's the way to constructively approach an article like this, out-of-hand dismissal is just dumb and unproductive.

FWIW I've seen the leaders of software teams all the way up to the CTO run around like headless chickens during (often self inflicted) crisis. I think the biggest lesson from the Titanic is that you're never invulnerable, even when you have been designed to be invulnerable.

None of these are exhaustive and all of them are open to interpretation. Good, so let's improve on them.

One general takeaway: managing risk is hard, especially when working with a limited budget (which is almost always the case) and just the exercise of assessing and estimating likelihood and impact are already very valuable but plenty of organizations have never done any of that. They simply are utterly blind to the risks their org is exposed to.

Case in point: a company that made in-car boxes that could be upgraded OTA. And nobody thought to verify that the vehicle wasn't in motion...

Re: Software engineering lessons from RCAs of greatest disasters

#63

Post by RetroTechie »

hliyan wrote: post #11

I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.

In terms of engineering (QC, processes etc), modern day software industry is worse than almost any other industry out there. :-(

And no, just plain complexity or fast-moving environment, is a factor but not the issue. It's that steps are skipped which are not skipped in other branches of engineering (eg. continous improvement of processes, learning from mistakes & implementing those. In software land: same mistakes made again & again & again, poorly designed languages remain in use, the list goes on).

A long way still to go.

Re: Software engineering lessons from RCAs of greatest disasters

#64

Post by jrockway »

GreenVulpine wrote: post #31
Exuma wrote: post #15

Earlier quoted context omitted.

My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments

You could call that a feature making payments easier for customers! All 3DS does is protect the banks by inconveniencing consumers since banks are responsible for fraud.

I believe they pass on the risk to merchants now. If you let fraud through, $30 per incident or whatever. So typically things like 3DS are turned on because that cost got too high, and the banks assure you that it will fix everything.

Re: Software engineering lessons from RCAs of greatest disasters

#65

Post by jacquesm »

RetroTechie wrote: post #63
hliyan wrote: post #11

I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.

In terms of engineering (QC, processes etc), modern day software industry is worse than almost any other industry out there. :-( And no, just plain complexity or fast-moving environment, is a factor but not the issue. It's that steps are skipped which are not skipped in other branches of engineering (eg. continous improvement of processes, learning from mistakes & implementing those. In software land: same mistakes m…

And reading this thread it doesn't look as if there is much awareness of that.

Re: Software engineering lessons from RCAs of greatest disasters

#66

Post by jacquesm »

rob74 wrote: post #33
embik wrote: post #18

Earlier quoted context omitted.

Not entirely true: > Dittmann could not find an emergency brake in the corridor and had not noticed that there was an emergency brake handle in his own compartment. The learning from that should maybe instead be to keep non-technical management out of engineering decisions. The Wikipedia article fails to mention there was a specific manager who pushed the new wheel design into production and then went on to have a lo…

The English article sounds more like he started looking for an emergency brake after he had notified the conductor (and apparently failed to convince him of the urgency of the situation), not before. The German article is much longer, but only mentions that both the passenger and the conductor could have prevented the accident if they would have pulled the emergency brake immediately, but that the conductor was actin…

In the movie 'Kursk' there is exactly such a scene and the literal quote is 'By the book, better start praying. I'm not reli...".

Re: Software engineering lessons from RCAs of greatest disasters

#67

Post by fredley »

RetroTechie wrote: post #63
hliyan wrote: post #11

I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.

In terms of engineering (QC, processes etc), modern day software industry is worse than almost any other industry out there. :-( And no, just plain complexity or fast-moving environment, is a factor but not the issue. It's that steps are skipped which are not skipped in other branches of engineering (eg. continous improvement of processes, learning from mistakes & implementing those. In software land: same mistakes m…

In any other field of engineering, the engineers are all trained and qualified. In software 'engineering', not so much.

Re: Software engineering lessons from RCAs of greatest disasters

#68

Post by LeonenTheDK »

fredley wrote: post #67
RetroTechie wrote: post #63

Earlier quoted context omitted.

In terms of engineering (QC, processes etc), modern day software industry is worse than almost any other industry out there. :-( And no, just plain complexity or fast-moving environment, is a factor but not the issue. It's that steps are skipped which are not skipped in other branches of engineering (eg. continous improvement of processes, learning from mistakes & implementing those. In software land: same mistakes m…

In any other field of engineering, the engineers are all trained and qualified. In software 'engineering', not so much.

This is exactly why I support "engineer" being a protected term, like Doctor. It should tell you that a certain level of training and qualification has been met, to the point that the engineer is responsible and accountable for the work they do and sign off on. Especially for things that affect safety.

Many software engineers these days are often flying by the seat of their pants, moving quickly and breaking things. Thankfully this seems to largely be in places that aren't going to affect life or limb, but I'm still rubbed the wrong way seeing people (including myself mind you) building run of the mill CRUD apps under the title of engineer.

Is it a big deal? Not really. It's probably even technically correct to use the term this way. But I do think it dilutes it a bit. For context, I'm in Canada where engineer is technically a protected term, and there are governing bodies that qualify and designate professional engineers.

Re: Software engineering lessons from RCAs of greatest disasters

#69

Post by lazystar »

ah, a topic related to organizational decay and decline. this is an area I've been studying a lot over the last few years, and I encourage the author of this blog post to read this paper on the Challenger disaster.

Organizational disaster and organizational decay: the case of the National Aeronautics and Space Administration http://www.sba.oakland.edu/faculty/schwartz/Org%20Decay%20at...

some highlights:

> There are a number of aspects of organizational decay. In this paper, I shall consider three of them. First is what I call the institutionalization of the fiction, which represents the redirection of its approved beliefs and discourse, from the acknowledgement of reality to the maintenance of an image of itself as the or­ganization ideal. Second is the change in personnel that parallels the institu­tionalization of the fiction. Third is the narcissistic loss of reality which repre­sents the mental state of management in the decadent organization.

> Discouragement and alienation of competent individuals

> Another result of this sort of selection must be that realistic and competent persons who are committed to their work must lose the belief that the organi­zation's real purpose is productive work and come to the conclusion that its real purpose is self-idealization. They then are likely to see their work as being alien to the purposes of the organization. Some will withdraw from the orga­nization psychologically. Others will buy into the nonsense around them, cyn­ically or through self-deception (Goffman, 1959), and abandon their concern with reality. Still others will conclude that the only way to save their self-esteem is to leave the organization. Arguably, it is these last individuals who, because of their commitment to productive work and their firm grasp of reality, are the most productive members of the organization. Trento cites a number of ex­amples of this happening at NASA. Considerations of space preclude detailed discussion here.

Schwartz, H.S., 1989. Organizational disaster and organizational decay: the case of the National Aeronautics and Space Administration. Industrial Crisis Quarterly, 3: 319-334.

Re: Software engineering lessons from RCAs of greatest disasters

#70

Post by serjester »

RetroTechie wrote: post #63
hliyan wrote: post #11

I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.

In terms of engineering (QC, processes etc), modern day software industry is worse than almost any other industry out there. :-( And no, just plain complexity or fast-moving environment, is a factor but not the issue. It's that steps are skipped which are not skipped in other branches of engineering (eg. continous improvement of processes, learning from mistakes & implementing those. In software land: same mistakes m…

If you're making the same mistakes over and over again I think that says more about your company than it does about the software industry.

My first job was at a major automotive manufacturer. Implementing half the procedures they had would slow down any software company 10X - just look at the state of most car infotainment systems. If something is safety critical, obviously this makes sense but the reality is 85% of software isn't.

Post reply on HN