Earlier quoted context omitted.
The global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots Wolves ate through fiber - 5 shots And it was a full moon - 6 shots Single service failure, but service has not been restarted in 5 years, and no lo…
5 is often broken down to “rain fade” and “backhoe fade.” My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
Here it is anyway: https://news.ycombinator.com/item?id=22316230