21–30 of 150 posts
Most of these feel like "They didnt act fast enough or do a good enough job. They should do a better job".
Another train crash that holds a valuable lesson was https://en.wikipedia.org/wiki/Eschede_train_disaster This demonstrates that sometimes "if you see something, say something" isn't enough - if a large piece of metal penetrates into the passenger compartment of a train from underneath, it's better to take the initiative and pull the emergency brake yourself .
Not entirely true: > Dittmann could not find an emergency brake in the corridor and had not noticed that there was an emergency brake handle in his own compartment. The learning from that should maybe instead be to keep non-technical management out of engineering decisions. The Wikipedia article fails to mention there was a specific manager who pushed the new wheel design into production and then went on to have a lo…
We could also easily construe your argument as "engineers would never design a flaw", which is demonstrably untrue. We should both work to minimize errors, and to provide a variety of corrective measures in case they happen.
Earlier quoted context omitted.
Not entirely true: > Dittmann could not find an emergency brake in the corridor and had not noticed that there was an emergency brake handle in his own compartment. The learning from that should maybe instead be to keep non-technical management out of engineering decisions. The Wikipedia article fails to mention there was a specific manager who pushed the new wheel design into production and then went on to have a lo…
I dont think it should be "instead". Suggesting that emergency brakes are inadequate due to one passenger failing to locate one is kinda cheap. We could also easily construe your argument as "engineers would never design a flaw", which is demonstrably untrue. We should both work to minimize errors, and to provide a variety of corrective measures in case they happen.
That’s not what I wanted to say at all - the op talked about the willingness to pull the emergency brake, but my understanding is that he was willing to but due to human error failed to find it. I didn’t mean to suggest in any way that emergency brakes are not important.
> We could also easily construe your argument as "engineers would never design a flaw"
Another thing I didn’t say. The whole original link is proof that engineers make mistakes all the time.
I don't think this list hits any fundamental truths. The great depression doesn't have parallels to software failures beyond the fact that complex systems fail. And many of the lessons are vague and unactionable - "Put an end to information hoarding within orgs/teams" for example, says nothing. The Atlassian copy that section links to also says nothing. A lot of the lessons lack meaty learnings, and good luck to anyo…
Complex systems fail, but they don't all fail in the same way and analyzing how they fail can help in engineering new and hopefully more robust complex systems. I'm a huge fan of Risk Digest and there isn't a disaster small enough that we can't learn from it. Obviously the larger the disaster the more complex the failure and the harder to analyze the root cause. But one interesting takeaway for me from this list is t…
Every disaster is preventable. Everything on the list was happening in human-engineered environments - as do most things that affect humans. The human race has been the master of its own destiny since the 1900s. The questions are how far before the disaster we need to look to find somewhere to act and what needed to be given up to change the flow of events.
But that doesn't have any implications for software engineering. Studying a software failure post mortem will be a lot more useful than studying 9/11.
[flagged]
If your point is that the person is likely to be ignored by his/her management, there's likely a better way to phrase it or worth adding a few words to clarify.
Earlier quoted context omitted.
My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments
Pretty epic. I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL. I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/
It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.
I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.
My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments
Earlier quoted context omitted.
Pretty epic. I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL. I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/
I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.