Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Complex Systems Fail (1998)

how.complexsystems.fail

Post reply on HN
Print view View original

51–60 of 90 posts

Re: How Complex Systems Fail (1998)

#51

Post by esafak »

cpgxiii wrote: post #49
esafak wrote: post #45

Earlier quoted context omitted.

> After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. What does that mean? If you knew that the precursors were why did you not set alerts?

The "naive notion of system performance" is that the system is primarily operating in the ideal regime, and thus all deviations deserve alerts (and receive attention). The reality is that often the system is operating often enough in some failure mode that any such alerts would be regularly triggered by accepted behavior, and thus the alerts are systematically ignored/disabled/normalized. E.g. nominally you should ne…

My response would be to use more sophisticated anomaly detection.

Re: How Complex Systems Fail (1998)

#52

Post by lstodd »

Vinnl wrote: post #30
jedberg wrote: post #24

Earlier quoted context omitted.

I agree with you. Kids aren't allowed to fail anymore. And as a parent, when I try to let my kid fail, I get scolded for being a bad parent. Broken bones used to be a right of passage for kids. Now it happens far less. While a broken bone sucks, it teaches you important lessons! Don't do that again, and hey, you messed up but you're fine now.

I feel like painfully falling without breaking bones are pretty good lessons already, and I don't quite see what the broken bones add there.

As someone who broke a wrist by rear-ending a car on my 100cc scooter I must respectfully disagree. A broken bone is way superior learning experience compared to some bruises.

Re: How Complex Systems Fail (1998)

#53

Post by onion2k »

pdonis wrote: post #34
tptacek wrote: post #16

I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing. The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure…

> "root cause analysis", at least on complex systems, is a fools errand. Not when properly understood--which, to be fair, it often isn't. The proper purpose of root cause analysis is to answer the question: does this failure mean I need to change something about this system? If the answer turns out to be yes, the thing that you need to change gets called the "root cause" of the failure. Of course that's a misnomer, s…

does this failure mean I need to change something about this system?

It's safer to say 'does this failure mean I need to change the system?'. Otherwise you're still facing the single cause trap.

Re: How Complex Systems Fail (1998)

#54

Post by zero_k »

This is very important work. If it interests you, and you want to dig further, it's broadly called "Safety II". You probably want to read Erik Hollnagel [1] or Sydney Dekker. I put together a presentation once about how this translates to IT security [2]. But broadly:

Safety I Learn from Errors, Safety defined by absence,Reactive approach, Understanding what goes wrong, Accident causation models, Avoidance of errors, Reducing losses

Safety II Learn from successes, Safety defined by presence, Proactive approach, Understanding what goes right, Repeat what goes right, Enforce successful behaviours, Create new processes based on successful behaviour

If anything, think long and hard about this: "Post-accident attribution to a ‘root cause’ is fundamentally wrong." It's a true statement. If you truly understand this sentence, you have mastered Safety II to the level that a light bulb will come on, and you won't be able to un-see it. This light bulb is a curse, a beautiful curse.

[1] https://safety4sea.com/cm-safety-i-vs-safety-ii-an-overview [2] https://www.msoos.org/largefiles/safety2.pdf

Re: How Complex Systems Fail (1998)

#55

Post by ChrisMarshallNY »

Whenever I see this [excellent] paper, I am always struck by the first sentence, in the first section:

> All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by THE own nature.

(Emphasis mine)

I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.

Re: How Complex Systems Fail (1998)

#56

Post by [deleted] »

esafak wrote: post #51
cpgxiii wrote: post #49

Earlier quoted context omitted.

The "naive notion of system performance" is that the system is primarily operating in the ideal regime, and thus all deviations deserve alerts (and receive attention). The reality is that often the system is operating often enough in some failure mode that any such alerts would be regularly triggered by accepted behavior, and thus the alerts are systematically ignored/disabled/normalized. E.g. nominally you should ne…

My response would be to use more sophisticated anomaly detection.

[deleted]

Re: How Complex Systems Fail (1998)

#57

Post by DJBunnies »

ChrisMarshallNY wrote: post #55

Whenever I see this [excellent] paper, I am always struck by the first sentence, in the first section: > All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by THE own nature. (Emphasis mine) I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.

Is it not merely an example of such failure?

Re: How Complex Systems Fail (1998)

#58

Post by pdonis »

onion2k wrote: post #53
pdonis wrote: post #34

Earlier quoted context omitted.

> "root cause analysis", at least on complex systems, is a fools errand. Not when properly understood--which, to be fair, it often isn't. The proper purpose of root cause analysis is to answer the question: does this failure mean I need to change something about this system? If the answer turns out to be yes, the thing that you need to change gets called the "root cause" of the failure. Of course that's a misnomer, s…

does this failure mean I need to change something about this system? It's safer to say 'does this failure mean I need to change the system?'. Otherwise you're still facing the single cause trap.

> Otherwise you're still facing the single cause trap.

"Something" does not have to mean "one single thing". But I would be fine with your phraseology if it ended up making it clearer to whoever was doing the analysis.

Re: How Complex Systems Fail (1998)

#59

Post by Angostura »

YZF wrote: post #25
tptacek wrote: post #16

I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing. The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure…

Root Cause Analysis is an organizational ritual that makes people feel like they're doing something. It's more about politics than about engineering. Not only is the process not useful- more often than not the correct root cause isn't even correctly identified. Basically a poorly designed system is going to fail somewhere. I like to think about it either as the part of the iceberg that's visible or my other analogy i…

A root cause investigation that concludes 'the system as a whole is fragile and poorly designed' sounds highly valuable - and they do happenl; I've seen them in healthcare

Re: How Complex Systems Fail (1998)

#60

Post by sandeepkd »

tptacek wrote: post #31
sandeepkd wrote: post #21

Overall a good collection on the complex systems, somehow it missed the part how complex system came into the existence in the first place itself. > Human expertise in complex systems is constantly changing I feel this is single most important factor responsible to both making system complex and at the same time improving them depending on who the people are and how they take failures and breakdowns. I find it funny…

Literally the first item in the list addresses the necessity of complexity in the systems it's discussing.

A hazardous system does not necessarily has to be complex. The article is a generalized list which makes it bit harder to reason with on this topic. A big part of the complexity gets added over the time as a part of failures, effort to introduce more fail safes, processes to avoid the same failure again. Complexity begets complexity, which is why I was referring to the dependency on the human elements who are involved in evaluating the failure and remedies.

tldr; Removing a process is risky and thankless for most part, no one wants to take that risk. So after every failure a new process gets added and thus you end up with an ever growing complex system

Post reply on HN