11–20 of 150 posts
F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2...
Loss of Mars probe due to metric-imperial conversion error
I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.
I don't think this list hits any fundamental truths. The great depression doesn't have parallels to software failures beyond the fact that complex systems fail. And many of the lessons are vague and unactionable - "Put an end to information hoarding within orgs/teams" for example, says nothing. The Atlassian copy that section links to also says nothing. A lot of the lessons lack meaty learnings, and good luck to anyo…
Obviously the larger the disaster the more complex the failure and the harder to analyze the root cause. But one interesting takeaway for me from this list is that all of them were preventable and in all but few of the cases the root cause may have been the trigger but the setup of the environment is what allowed the fault to escalate in the way that it did. In a resilient system faults happen as well, but they do not propagate.
And that's the big secret to designing reliable systems.
(Root Cause Analysis)
This whole RCA terminology has to die. The idea that there exists a single "Root Cause" that causes major disasters is fundamentally flawed.
I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.
Earlier quoted context omitted.
This whole RCA terminology has to die. The idea that there exists a single "Root Cause" that causes major disasters is fundamentally flawed.
Actually everywhere "defense in depth" is used (not only in computing, but also in e.g. aviation), there can't be one single cause for a disaster - each of the layers has to fail for a disaster to happen.
In software the root cause is often a very simple one: assumption didn't hold.
I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.
My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments
I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL.
I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/
Another train crash that holds a valuable lesson was https://en.wikipedia.org/wiki/Eschede_train_disaster This demonstrates that sometimes "if you see something, say something" isn't enough - if a large piece of metal penetrates into the passenger compartment of a train from underneath, it's better to take the initiative and pull the emergency brake yourself .
> Dittmann could not find an emergency brake in the corridor and had not noticed that there was an emergency brake handle in his own compartment.
The learning from that should maybe instead be to keep non-technical management out of engineering decisions. The Wikipedia article fails to mention there was a specific manager who pushed the new wheel design into production and then went on to have a long successful career.
I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.
My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments
I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.