How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…
From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); }…
This is really interesting and something which bugs me about root cause analysis and it's a neat coincidence that this has been quoted relative to an aviation incident.
In aviation, incidents and accidents are investigated with the understanding that there is never a single cause of an accident. It's known as the swiss cheese model. All the holes in the swiss cheese have to line up for something to go wrong. Even in a seemingly simple "pilot error" accident, there are years of initial and recurrent training factors, ergonomic and human factors and so on which all lead to the event. It's exceedingly rare for a single "root cause" to be the whole story.
Medicine is starting to adopt techniques learned from aviation like checklists, crew resource management and no-blame, swiss-cheese accident investigations. I am hopeful that the software industry will take similar lessons over the next decade or so.