Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software engineering lessons from RCAs of greatest disasters

anoopdixith.com

Post reply on HN
Print view View original

141–150 of 150 posts

Re: Software engineering lessons from RCAs of greatest disasters

#141

Post by jacquesm »

KnobbleMcKnees wrote: post #140
jacquesm wrote: post #87

Earlier quoted context omitted.

It could easily be from looking from the outside in, as it is in my case.

Oh, so, without experience. Understood.

No, with lots of experience. I've been programming since 17, did it professionally for three decades and have since moved into technical due diligence, 16 years and counting. That gives me a pretty unique perspective on IT, I get to talk to and work with teams from a very large sample of companies (230+ to date) and that in turn gives a fairly large statistical base to base that opinion on. This includes seed stage companies, mid sized and very large ones.

So forgive me if I take your attitude as non-productive, you are essentially just trying to discount my input based on some assumptions because it apparently doesn't please you. I'm fine with that but then just keep it to yourself instead of pulling down the level of discourse here. If you wanted to make a point you could have been constructive rather than dismissive.

Re: Software engineering lessons from RCAs of greatest disasters

#142

Post by seanhunter »

jacquesm wrote: post #109
seanhunter wrote: post #104

Earlier quoted context omitted.

Yeah maybe. I think that sort of rings a bell.

https://youtu.be/hcIgZ4kJ5Ow?t=340 this?

That is exactly it. For some reason my brain had spliced Schwartenegger into the cockpit. Funny how memory works.

Re: Software engineering lessons from RCAs of greatest disasters

#143

Post by jacquesm »

seanhunter wrote: post #142
jacquesm wrote: post #109

Earlier quoted context omitted.

https://youtu.be/hcIgZ4kJ5Ow?t=340 this?

That is exactly it. For some reason my brain had spliced Schwartenegger into the cockpit. Funny how memory works.

It really threw me because I also remembered the scene but not the actor so I kept looking for Schwarzenegger movies. Good, checked that one off the list :)

Re: Software engineering lessons from RCAs of greatest disasters

#144

Post by yafbum »

nvy wrote: post #139
yafbum wrote: post #138

Earlier quoted context omitted.

I'm not saying just write the function and flash the firmware, but it's not like the super rigid certification process doesn't have its nefarious side effects either. My experience is that the more expensive fixes are, the more humans are willing to turn a blind eye to problems or wish them away.

>but it's not like the super rigid certification process doesn't have its nefarious side effects either. The system isn't rigid so much as thorough. You can omit portions of the review for Minor Changes (term of art), for example. Unfortunately "writing the code to correctly release deadly explosives from the aircraft in flight" is far from a Minor Change, so gee willikers I guess it required some due diligence. Mayb…

I hear you. I respect the process and practice. I would invite you to ponder what would happen if all the iPhones and iPads in one country were to be bricked overnight by an OTA update - billions of dollars worth of instant economic damage just from the device cost, many billions more in consequences including lives lost. Well this capability probably exists somewhere at Apple. I hope it's well guarded by process and perhaps this process is costly not unlike recertification. Does the ability to deliver critical fixes quickly make it a safer system on balance, versus the Nokia and BlackBerry era where your firmware essentially never changed ever because the cost of delivery was so high? My guess it that it does on balance represent an improvement. But maybe I misunderstood and the millions of dollars in cost of delivering the fix were actually spent on due diligence, as opposed to just mechanically applying the patch.

Re: Software engineering lessons from RCAs of greatest disasters

#145

Post by chupy »

Two4 wrote: post #115
nvy wrote: post #90

Earlier quoted context omitted.

>It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach A bunch of JavaScript dudebros yeeting code out into the ether is not at all comparable to deploying avionics software to a fighter jet. Give your head a shake.

I don't think they're referring to dudebros' js, they're referring to systems software and the ability to deliver relatively secure updates over insecure channels. I've even delivered a signed firmware update to a microprocessor in a goddamn washing machine over UART. Why can't we do this for a jet?

Is it because you risk doing something like this? https://hackaday.com/2022/03/18/welcome-to-the-future-where-...

Re: Software engineering lessons from RCAs of greatest disasters

#146

Post by el_toast »

yafbum wrote: post #144
nvy wrote: post #139

Earlier quoted context omitted.

>but it's not like the super rigid certification process doesn't have its nefarious side effects either. The system isn't rigid so much as thorough. You can omit portions of the review for Minor Changes (term of art), for example. Unfortunately "writing the code to correctly release deadly explosives from the aircraft in flight" is far from a Minor Change, so gee willikers I guess it required some due diligence. Mayb…

I hear you. I respect the process and practice. I would invite you to ponder what would happen if all the iPhones and iPads in one country were to be bricked overnight by an OTA update - billions of dollars worth of instant economic damage just from the device cost, many billions more in consequences including lives lost. Well this capability probably exists somewhere at Apple. I hope it's well guarded by process and…

Many people will hear the usual story of the fixes (for the plane example) being enormously expensive without really diving into what all goes into that figure. The source code change itself may be trivial, so it's easy to compare that to the multi-million dollar figures thrown around and have criticisms.

We can do OTA updates, there's no technical reason it can't be done other than not allowing it (which I mostly agree with in secure applications). Hell our spacecraft do this now. We must keep in mind these fixes do not go from dev environment straight to the field (prod), which would be a terrible idea. These are extremely complex integrated systems and must be tested in multiple phases because let's face it, if this supposedly trivial issue made it all the way through, what else may not have been discovered yet?

Not only does the 'easy' fix need to be tested (time and money), but related interactions need to be investigated as well (more time and money). The time cost of people doing the work, investigations, testing adds up from all this. Then there's potentially hardware in the mix which is never cheap, also simply being able to get access to hardware for testing can be a huge hassle.

Keep in mind this comment is only geared towards situations where the end item is a physical system. I would expect a fixing a pure software product to have significantly lower costs.

Re: Software engineering lessons from RCAs of greatest disasters

#147

Post by twic »

Two4 wrote: post #115
nvy wrote: post #90

Earlier quoted context omitted.

>It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach A bunch of JavaScript dudebros yeeting code out into the ether is not at all comparable to deploying avionics software to a fighter jet. Give your head a shake.

I don't think they're referring to dudebros' js, they're referring to systems software and the ability to deliver relatively secure updates over insecure channels. I've even delivered a signed firmware update to a microprocessor in a goddamn washing machine over UART. Why can't we do this for a jet?

Because your firmware would put the jet into a spin.

Re: Software engineering lessons from RCAs of greatest disasters

#148

Post by twic »

jacquesm wrote: post #109
seanhunter wrote: post #104

Earlier quoted context omitted.

Yeah maybe. I think that sort of rings a bell.

https://youtu.be/hcIgZ4kJ5Ow?t=340 this?

Oh, that's also the sequence with the "peace / war" switch! That did make me laugh. Turns out it's a real thing, though - but they probably wouldn't have flipped it in this situation.

Re: Software engineering lessons from RCAs of greatest disasters

#149

Post by wlonkly »

gdevenyi wrote: post #26

This is why software engineering is a protected profession in some parts of the world (Canada at least), as civil responsibility and safety, along with formal legal liability is part of licensure

Technically true, but most software developers in Canada aren't P. Engs, and tons of Canadian software companies use a "Software Engineer" title with no repercussions, so I'm not sure you can point at that as a success.

Re: Software engineering lessons from RCAs of greatest disasters

#150

Post by KnobbleMcKnees »

jacquesm wrote: post #141
KnobbleMcKnees wrote: post #140

Earlier quoted context omitted.

Oh, so, without experience. Understood.

No, with lots of experience. I've been programming since 17, did it professionally for three decades and have since moved into technical due diligence, 16 years and counting. That gives me a pretty unique perspective on IT, I get to talk to and work with teams from a very large sample of companies (230+ to date) and that in turn gives a fairly large statistical base to base that opinion on. This includes seed stage c…

[flagged]
Post reply on HN