Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Complex Systems Fail (1998)

how.complexsystems.fail

Post reply on HN
Print view View original

41–50 of 90 posts

Re: How Complex Systems Fail (1998)

#41

Post by ErroneousBosh »

tptacek wrote: post #27
ErroneousBosh wrote: post #19

Earlier quoted context omitted.

> I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing. If you want a bit of cheese to go with that wine, this article pairs nicely with The Grug-Brained Developer: https://grugbrain.dev/

I am also a broken record about Grug-brainedness, but I don't need to be so repetitive about it because so many other people are too.

My Grug-brained hill to die on is that there are many pieces of technology like the LM741 opamp, the Rover V8, Morris A-series and Peugeot XUD engines, and a big SQL database running on a thumping great Unix box, that have remained essentially unchanged for decades, worked well when they were new, and are still a perfectly good solution today for so much of what you want to do.

You know what they have in common? A lack of complexity. Lovely simple technology, fix it with a hammer. Drive it to the moon and back. Drive it across the desert. Bring tools though!

Re: How Complex Systems Fail (1998)

#42

Post by anonymars »

"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously."

This very much resembles Admiral Cloudberg's write-up of the National Airport collision:

"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."

https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...

Re: How Complex Systems Fail (1998)

#43

Post by [deleted] »

anonymars wrote: post #42

"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions…

[deleted]

Re: How Complex Systems Fail (1998)

#44

Post by jonahx »

jasode wrote: post #38
tptacek wrote: post #16

I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing. The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure…

>The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. It's only an issue when people take that phrase very literally. People have common sense to understand that things have multiple causes and a chain of events. NASA has "Root Cause Analysis" (singular) all over various official documentation and it doesn't stop them from understanding that th…

> Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations.

In the case of Challenger, I think it's pretty clear this was the "root cause":

> According to testimony by Kilminster and Boisjoly, Mason finally turned to Bob Lund and said, "Take off your engineering hat and put on your management hat." Joe Kilminster wrote out the new recommendation and went back on line with the teleconference.

> The new recommendation stated that the cold was still a safety concern, but their people had found that the original data was indeed inconclusive and their "engineering assessment" was that launch was recommended, even though the engineers had no part in writing the new recommendation and refused to sign it.

-- https://onlineethics.virginia.edu/cases/engineering-ethics-c...

If you want to take the "system" view here, as is often the case, it is the organizational power structure and incentives therein that comprise the dangerous system. You had engineering experts easily predicting the disaster, but they had no decision making power. That was the problem. But if you set up an organization like that, where the egos of "get it done" managers are allowed to gamble with other people's lives to win their own accolades, the system is doomed from the start.

Re: How Complex Systems Fail (1998)

#45

Post by esafak »

anonymars wrote: post #42

"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions…

> After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance.

What does that mean? If you knew that the precursors were why did you not set alerts?

Re: How Complex Systems Fail (1998)

#46

Post by asenchi »

tptacek wrote: post #16

I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing. The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure…

Same, I think this document is fundamental to our industry and understanding complex systems. We have it as a quarterly reminder to review in our Slack. I give it to everyone that makes it past the first couple of interviews when applying and tout its usefulness for thinking through systems design all the time. RIP Dr. Cook.

Re: How Complex Systems Fail (1998)

#47

Post by obscurette »

eskimobloood wrote: post #23
obscurette wrote: post #18

Earlier quoted context omitted.

It's much more universal and complicated than that. One of the big issues schools and pedagogy in general struggle with is that our environment is far too safe in too many ways. For kids there is too few ways to learn from failures. Attempts to solve the issue look often like "Hey, kids, let's fall over now all at once in safest way possible and learn from it!". But it doesn't work at all. Real failures have to be un…

> One of the big issues schools and pedagogy in general struggle with is that our environment is far too safe in too many ways. For kids there is too few ways to learn from failures. Its cause we already learned from our failures and make the world a safer place. In my youth we climbed on trees, one of my friend fall down broke his arm, the doctor couldn't fix it and he can not move the hand for the rest of his live.…

No. I'm making a claim that the world is too safe from a perspective of my students. I see it very clearly that a lot of anxiety actually originates from fear. They are afraid to explore environment because they are not allowed to do it. They are afraid of conflicts because they are not allowed to learn to resolve these themselves. They are afraid of resolve any of their problems themselves because they are not allowed to discover the power to do it etc.

Re: How Complex Systems Fail (1998)

#48

Post by marcosdumay »

DrewADesign wrote: post #35
pdonis wrote: post #34

Earlier quoted context omitted.

> "root cause analysis", at least on complex systems, is a fools errand. Not when properly understood--which, to be fair, it often isn't. The proper purpose of root cause analysis is to answer the question: does this failure mean I need to change something about this system? If the answer turns out to be yes, the thing that you need to change gets called the "root cause" of the failure. Of course that's a misnomer, s…

I don’t blame people for being confused about the term in software — it’s a bit more straightforward in manufacturing, for example. That said, if you look at a system that failed and determine a root cause analysis is pointless, you may have just stumbled upon your root cause.

> it’s a bit more straightforward in manufacturing

That's because manufacturing is a simpler environment where causes and consequences are more conceptually distinct.

Until you start to pull management decisions into your analysis, of course. Then it becomes just as complex as software.

Re: How Complex Systems Fail (1998)

#49

Post by cpgxiii »

esafak wrote: post #45
anonymars wrote: post #42

"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions…

> After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. What does that mean? If you knew that the precursors were why did you not set alerts?

The "naive notion of system performance" is that the system is primarily operating in the ideal regime, and thus all deviations deserve alerts (and receive attention). The reality is that often the system is operating often enough in some failure mode that any such alerts would be regularly triggered by accepted behavior, and thus the alerts are systematically ignored/disabled/normalized.

E.g. nominally you should never be mixing traffic types (aircraft and helicopters, civilian and military) in close proximity to a major airport and in a regime where TCAS is unlikely to offer sufficient protection. So in theory, any mixing should immediately trigger an alert and investigation to develop new procedures. But in practice, if you routinely allow such mixing under what you believe are "safe" practices (and get away with such mixing for a long time) then when a real accident happens you will have plenty of "proto-accidents" to look back on, but the warning signs from those near-accidents will have become accepted practice.

Re: How Complex Systems Fail (1998)

#50

Post by pdonis »

marcosdumay wrote: post #48
DrewADesign wrote: post #35

Earlier quoted context omitted.

I don’t blame people for being confused about the term in software — it’s a bit more straightforward in manufacturing, for example. That said, if you look at a system that failed and determine a root cause analysis is pointless, you may have just stumbled upon your root cause.

> it’s a bit more straightforward in manufacturing That's because manufacturing is a simpler environment where causes and consequences are more conceptually distinct. Until you start to pull management decisions into your analysis, of course. Then it becomes just as complex as software.

> Until you start to pull management decisions into your analysis, of course.

Meaning, having something like "management will decide not to fund maintenance for this component in the future, causing a failure" in your fault tree?

Post reply on HN