Software engineering lessons from RCAs of greatest disasters • Hacker News Board

Re: Software engineering lessons from RCAs of greatest disasters

#41

Post by jacquesm » Thu, Aug 17, 2023, 1:47 PM UTC

CTDOCodebases wrote: post #28

ownagefool wrote: post #17

Earlier quoted context omitted.

Pretty epic. I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL. I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/

I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.

I if any such pathways remain at AWS, Google, Apple and MS that would still allow a thing like that to happen.

Re: Software engineering lessons from RCAs of greatest disasters

#42

Post by jacquesm » Thu, Aug 17, 2023, 1:50 PM UTC

swalsh wrote: post #29

Exuma wrote: post #15

Earlier quoted context omitted.

My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments

My worst bug was changing how a zip code zone was fetched from the cache in a large ecommerce site with tens of thousands users using it all day long. Worked great in DEV :D but when the thundering herd hit it, the entire site came down.

Startup, shutdown and migration are all periods of significantly elevated risk. Especially for systems that have been in the air for a long time there are all kinds of ways in which things can go pear shaped. Drives that die on shutdown (or the subsequent boot up), raids that fail to rebuild, cascading failures, power supplies that fail, UPS's that fail, generators that don't start (or that run for 30 seconds and then quit because someone made off with the fuel) and so on.

Re: Software engineering lessons from RCAs of greatest disasters

#43

Post by MisterTea » Thu, Aug 17, 2023, 1:52 PM UTC

seanhunter wrote: post #38

One of my favourites that I ever heard about was from a friend of mine who used to work on safety-critical systems in defense applications. He told me fighter jets have a safety system that disables the weapons systems if a (weight) load is detected on the landing gear so that if the plane is on the ground and the pilot bumps the wrong button they don't accidentally blow up their own airbase[1]. So anyway when the Eu…

> weird version of ADA

Spark: https://en.wikipedia.org/wiki/SPARK_(programming_language)

And it's Ada not ADA which makes me think of the Americans with Disabilities Act.

Re: Software engineering lessons from RCAs of greatest disasters

#44

Post by yafbum » Thu, Aug 17, 2023, 1:55 PM UTC

As I heard one engineering leader say, "it's okay to make mistakes — once". Meaning, we're all fallible, mistakes happen, but failure to learn from past mistakes is not optional.

That said, a challenge I have frequently run into, and I feel is not uncommon, is a tension between the desire not to repeat mistakes and ambitions that do generally involve some amount of risk-taking. The former can turn into a fixation on risk and risk mitigation that becomes a paralyzing force; to some leaders, lists like these might just look like "a thousand reasons to do nothing" and be discarded. Yet history is full of clear cases where a poor appreciation of risk destroyed fortunes, with a chorus of "I told you sos" in their wake.

It is a difficult part of leadership to weigh the risk tradeoffs for a particular mission, and presenting things in absolute terms of "lessons learned" rarely makes sense, in my experience. The belt-and-suspenders approaches that make sense for authoring the critical control software for a commercial passenger aircraft or an industrial control system for nuclear plants probably do not make sense for an indie mobile game studio, even if they're all in some way "software engineering".

Re: Software engineering lessons from RCAs of greatest disasters

#45

Post by yafbum » Thu, Aug 17, 2023, 2:03 PM UTC

seanhunter wrote: post #38

One of my favourites that I ever heard about was from a friend of mine who used to work on safety-critical systems in defense applications. He told me fighter jets have a safety system that disables the weapons systems if a (weight) load is detected on the landing gear so that if the plane is on the ground and the pilot bumps the wrong button they don't accidentally blow up their own airbase[1]. So anyway when the Eu…

> this cost several millions to redeploy the (one-line) fix to actually check the weight from the sensor was less than the threshold

Well maybe this is the other, compounding problem. Engineering complex machines with such a high cost of bugfix deployment seems like a big issue. It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach...

Or maybe, a few millions is like only a thousand hours of flying in jet fuel costs alone, not a big deal...

Re: Software engineering lessons from RCAs of greatest disasters

#46

Post by seanhunter » Thu, Aug 17, 2023, 2:07 PM UTC

MisterTea wrote: post #43

seanhunter wrote: post #38

One of my favourites that I ever heard about was from a friend of mine who used to work on safety-critical systems in defense applications. He told me fighter jets have a safety system that disables the weapons systems if a (weight) load is detected on the landing gear so that if the plane is on the ground and the pilot bumps the wrong button they don't accidentally blow up their own airbase[1]. So anyway when the Eu…

> weird version of ADA Spark: https://en.wikipedia.org/wiki/SPARK_(programming_language) And it's Ada not ADA which makes me think of the Americans with Disabilities Act.

Aah thank you on both counts yes. One interesting feature he told me about is they wrote a "reverse compiler" that would take Spark code and turn it into the associated formal (Z) spec so they could compare that to the actual Z spec to prove they were the same. Kind of nifty.

Re: Software engineering lessons from RCAs of greatest disasters

#47

Post by epolanski » Thu, Aug 17, 2023, 2:08 PM UTC

swalsh wrote: post #30

CTDOCodebases wrote: post #28

Earlier quoted context omitted.

I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.

At this point there's basically 3 clouds, and then everyone else.

AWS, Azure and Cloudflare?

Re: Software engineering lessons from RCAs of greatest disasters

#48

Post by tonyarkles » Thu, Aug 17, 2023, 2:10 PM UTC

BiggusDijkus wrote: post #5

formerly_proven wrote: post #3

(Root Cause Analysis)

This whole RCA terminology has to die. The idea that there exists a single "Root Cause" that causes major disasters is fundamentally flawed.

I think if the people you're working with insist on narrowing it down to a single Root Cause, they're missing the entire point of the exercise. I work with large drones day to day and when we do an accident investigation we're always looking for root causes, but there's almost always multiple. I don't think we've ever had a post-accident RCA investigation that resulted in only one corrective action. Several times we have narrowed it down to a single software bug, but to get to the point where a software bug causes a crash, there's always a number of other factors that have to align (e.g. pilot was unfamiliar with the recovery procedure, multiple cascaded failures, etc)

Re: Software engineering lessons from RCAs of greatest disasters

#49

Post by pc86 » Thu, Aug 17, 2023, 2:11 PM UTC

yafbum wrote: post #44

As I heard one engineering leader say, "it's okay to make mistakes — once ". Meaning, we're all fallible, mistakes happen, but failure to learn from past mistakes is not optional. That said, a challenge I have frequently run into, and I feel is not uncommon, is a tension between the desire not to repeat mistakes and ambitions that do generally involve some amount of risk-taking. The former can turn into a fixation on…

Mistakes happen, things go wrong, for the vast majority of us a bug doesn't mean someone dies or lights go out or planes don't take off. For most of us here, the absolute worst case scenario is that a bug means a company nobody has ever heard of makes slightly less money for a few minutes or hours until it gets rolled back. Again, worst case. The average case is probably closer to a company nobody has ever heard of makes exactly the same amount of money but some arbitrary feature nobody asked for ships a day or two later because we spent time fixing this other thing instead.

It's really hard to drive a balance between "pushing untested shitcode into prod multiple times a week" and "that ticket to change our CTA button color is done but now needs to go through 4 days of automated and manual testing." I think as an industry most of us are probably too far on the latter side of the spectrum in relation to the stakes of what we're actually doing day to day.

Re: Software engineering lessons from RCAs of greatest disasters

#50

Post by krisoft » Thu, Aug 17, 2023, 2:16 PM UTC

roenxi wrote: post #39

jacquesm wrote: post #32

Earlier quoted context omitted.

> Every disaster is preventable. No, there is such a thing as residual risk and there are always disasters that you can't prevent such as natural disasters. But even then you can have risk mitigation and strategies for dealing with the aftermath of an incident to limit the effects. > Everything on the list was happening in human-engineered environments - as do most things that affect humans. That is precisely why the…

It is 2023. The damage of natural disasters can be mitigated. When the San Andreas fault goes it'll probably get an entry on that list with a "why did we build so much infrastructure on this thing? Why didn't we prepare more for the inevitable?". And this article is throwing out generic all-weather good sounding platitudes which are tangential to the disasters listed. He drew a comparison between the Challenger disas…

> It is 2023. The damage of natural disasters can be mitigated.

That is a comforting belief, but it is probably not true. We have no plan for a near-Earth supernova explosion. Not even in theory.

Then there are asteroid impacts. In theory we could have plowed all of our resources into planetary defences, but in practice in 2023 we can very easily get sucker punched by a bolide and go the way of the dinosaurs.