Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software engineering lessons from RCAs of greatest disasters

anoopdixith.com

Post reply on HN
Print view View original

131–140 of 150 posts

Re: Software engineering lessons from RCAs of greatest disasters

#131

Post by CTDOCodebases »

jiscariot wrote: post #83
CTDOCodebases wrote: post #28

Earlier quoted context omitted.

I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.

That is eerily similar to what happened to us in IBM "Cloud", in a previous gig. An engineer was doing "account cleanup" and somehow our account got on the list and all our resources were blown away. The most interesting conversation was convincing the support person, that those deletion audit events were in fact not us, but rather (according to the engineer's Linked-In page) an SRE at IBM.

Although it probably wasn't funny at the time I can imagine how comical that conversation was.

Thinking about it further the term "cloud" is a good metaphor for storing files on someone else's computer because clouds just disappear.

Re: Software engineering lessons from RCAs of greatest disasters

#132

Post by masfuerte »

jacquesm wrote: post #100
masfuerte wrote: post #78

There is or was an old-school website called something like "Byzantine failures" that had case studies of bizarre failures from many engineering fields. It was entertaining but I am unable to find it now. Does anyone know it?

I think you're talking about Risks Digest. http://catless.ncl.ac.uk/Risks/ It's very much old-school.

Thank you, that is the site. I don't know where I got "Byzantine failures" from.

Re: Software engineering lessons from RCAs of greatest disasters

#133

Post by uxp100 »

Two4 wrote: post #115
nvy wrote: post #90

Earlier quoted context omitted.

>It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach A bunch of JavaScript dudebros yeeting code out into the ether is not at all comparable to deploying avionics software to a fighter jet. Give your head a shake.

I don't think they're referring to dudebros' js, they're referring to systems software and the ability to deliver relatively secure updates over insecure channels. I've even delivered a signed firmware update to a microprocessor in a goddamn washing machine over UART. Why can't we do this for a jet?

We don’t really know the context of this anecdote, but if you have to completely re-run your test plan on a real plane with real munitions for newly deployed software, which is a pretty good idea, then I could see it costing millions, even if the fix deployed in a minute.

Re: Software engineering lessons from RCAs of greatest disasters

#134

Post by jacquesm »

masfuerte wrote: post #132
jacquesm wrote: post #100

Earlier quoted context omitted.

I think you're talking about Risks Digest. http://catless.ncl.ac.uk/Risks/ It's very much old-school.

Thank you, that is the site. I don't know where I got "Byzantine failures" from.

No idea either, but the 'old school' hint did the trick. It is one of my favorite internet hang-outs. It's either that or HN (or reading books or playing piano). Endless interesting material and very useful for me professionally.

Re: Software engineering lessons from RCAs of greatest disasters

#136

Post by nvy »

Two4 wrote: post #115
nvy wrote: post #90

Earlier quoted context omitted.

>It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach A bunch of JavaScript dudebros yeeting code out into the ether is not at all comparable to deploying avionics software to a fighter jet. Give your head a shake.

I don't think they're referring to dudebros' js, they're referring to systems software and the ability to deliver relatively secure updates over insecure channels. I've even delivered a signed firmware update to a microprocessor in a goddamn washing machine over UART. Why can't we do this for a jet?

>Why can't we do this for a jet?

Well, because the software load of an aircraft is certified as part of the approved type design, for one. If you update the software it requires an engineering approval, because the risks inherent to operating an aircraft and the engineering that goes into mitigating those risks and making them acceptably safe are quite a bit more significant than a washing machine.

What's more we're talking about stores clearance (i.e. releasing shit from the aircraft in flight).

The attitude behind "Just write that function and flash the firmware" gets people killed.

Re: Software engineering lessons from RCAs of greatest disasters

#137

Post by wlindley »

Yet again another page that doesn't bother to explain what this has to do with the Radio Corporation of America, or what "RCA" might otherwise stand for. RCA's greatest disaster was Selectavision, but that's not on the list, and Wikipedia's disambiguation page doesn't have anything with anything relevant for what "RCA" might mean in relation to the things on this page. Explain yourselves, folks!

Re: Software engineering lessons from RCAs of greatest disasters

#138

Post by yafbum »

nvy wrote: post #136
Two4 wrote: post #115

Earlier quoted context omitted.

I don't think they're referring to dudebros' js, they're referring to systems software and the ability to deliver relatively secure updates over insecure channels. I've even delivered a signed firmware update to a microprocessor in a goddamn washing machine over UART. Why can't we do this for a jet?

>Why can't we do this for a jet? Well, because the software load of an aircraft is certified as part of the approved type design, for one. If you update the software it requires an engineering approval, because the risks inherent to operating an aircraft and the engineering that goes into mitigating those risks and making them acceptably safe are quite a bit more significant than a washing machine. What's more we're…

I'm not saying just write the function and flash the firmware, but it's not like the super rigid certification process doesn't have its nefarious side effects either. My experience is that the more expensive fixes are, the more humans are willing to turn a blind eye to problems or wish them away.

Re: Software engineering lessons from RCAs of greatest disasters

#139

Post by nvy »

yafbum wrote: post #138
nvy wrote: post #136

Earlier quoted context omitted.

>Why can't we do this for a jet? Well, because the software load of an aircraft is certified as part of the approved type design, for one. If you update the software it requires an engineering approval, because the risks inherent to operating an aircraft and the engineering that goes into mitigating those risks and making them acceptably safe are quite a bit more significant than a washing machine. What's more we're…

I'm not saying just write the function and flash the firmware, but it's not like the super rigid certification process doesn't have its nefarious side effects either. My experience is that the more expensive fixes are, the more humans are willing to turn a blind eye to problems or wish them away.

>but it's not like the super rigid certification process doesn't have its nefarious side effects either.

The system isn't rigid so much as thorough. You can omit portions of the review for Minor Changes (term of art), for example. Unfortunately "writing the code to correctly release deadly explosives from the aircraft in flight" is far from a Minor Change, so gee willikers I guess it required some due diligence.

Maybe sometimes doing things correctly takes time and money for a reason, even if the reason isn't obvious. Maybe there's a good reason not to have OTA firmware update capability on a warplane.

Re: Software engineering lessons from RCAs of greatest disasters

#140

Post by KnobbleMcKnees »

jacquesm wrote: post #87
KnobbleMcKnees wrote: post #86

Earlier quoted context omitted.

Is that not coming from experience of working at a software company? As I believe you said elsewhere

It could easily be from looking from the outside in, as it is in my case.

Oh, so, without experience. Understood.
Post reply on HN