How Complex Systems Fail (1998)

I've always struggled to apply this to the systems I work on. If the system fails, someone potentially dies, though in practice they've never been more than hospitalized. To avoid that, huge amounts of effort are expended on failure modelling and testing, but that doesn't eliminate unknown unknowns. That discrepancy has made front page news a couple times.

[dead]

I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.

The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more.

But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up.

All practitioner actions are gambles. I should have that inked somewhere.

It's much more universal and complicated than that. One of the big issues schools and pedagogy in general struggle with is that our environment is far too safe in too many ways. For kids there is too few ways to learn from failures. Attempts to solve the issue look often like "Hey, kids, let's fall over now all at once in safest way possible and learn from it!". But it doesn't work at all. Real failures have to be unexpected, related to your decisions and really hurt so that you can learn from them.

PS. Btw, I am certain that this is the main cause of the mental health crisis amongst young people.

One of the great documents of our civilisation