Earlier quoted context omitted.
Not really germane to the topic. This type of op fuck-up happens everywhere. It's hard to build solid process, particularly in growth phases. Unless there are 2x a year restore tests, I personally assume a 60% backup fail rate.
"It's hard to build solid process, particularly in growth phases." Hard is an understatement, it is insanely difficult. However, the interview seems to suggest that "write everything down" is the way to solve this problem and was what allowed for all their growth and success. So the solid process they have built is to write everything down, which they obviously didn't do or they wouldn't have had a 7 layer disaster r…
First, no one has the discipline to write much of anything down.
It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their peers will feel that they are dead weight, which is still a ticket out of the company.
Second, after all that work, no has the discipline to read much of anything that you've written!
They skim. They glance. They Ctrl+F. They don't read. When you're in a pickle and you can pick out a life-saving bit of documentation, it's amazing, but that happens quite rarely and requires a lot of energy.
How many times have you pulled up docs, tried to follow them, gotten a really confusing error that you spent hours trying to troubleshoot only to find out that there's a one-sentence explanation tucked away in the third sentence of the fourth paragraph on the page you originally pulled up? This just happened to me _last week_, and frequently the most frustrating problems are small things like that.
People don't read. It's nothing personal, they just don't read. It takes a lot of cognitive energy. People are biologically programmed to conserve as much as energy as possible. Good programmers are both lazy and dumb!
If you want documentation that means something, it needs to be part of the process of actually working. I don't mean you need to add "write docs" to your checklist, I mean meeting the operational standards should be the only way things can get done in the first place.
The operating procedure needs to be married to the actual completion of the task, and that means setting good baseline project standards and setting reliable enforcement on those standards.
Code should be self-documenting to a reasonable extent. Tests should be mandatory. Peer reviews and signoffs should be mandatory. Internal company discussions should be recorded and referenceable. Documentation only works when it's self-generating.
In short, it should be run like a mature open-source project with an open IRC channel, mailing list, bug tracker, commit history, mandatory tests, maintainer signoffs, merge processes, code and style standards, docstrings and good automated documentation generators, and so on.