Keep calm and use the runbook

> But how do you foster a culture in which everyone keeps it up to date?

Every service has a runbook. To ensure that they're kept up to date, teams are required to do twice-yearly Disaster Recovery (DR) drills where they fail their service out of one region or data center and into another. (This is for each service. They can be rescheduled - modulo some nagging - if absolutely necessary, but you still need to do them eventually as a part of OKRs / KTLO.)

DR exercises by necessity require that the runbook is up to date and that capacity measures and load testing are recent and accurate. By doing these semi-regularly, your team retains full familiarity with the operational characteristics of the system. Your service, your data stores, your dependencies, and your dependents. Hopefully you (or adjacent teams) won't discover any unknowns, but if you do, they also belong in a runbook or postmortem with action items set to address the issues.

Furthermore, every team keeps a weekly "oncall log", which is essentially a human aggregated and summarized log of pages received. The oncall engineer logs anything that crops up, including new failure classes, and triages fixing them (or gets assistance if they're junior). Novel happenings can easily be discussed during oncall handoff or briefly during standup. Some of these items will naturally find their way into the relevant runbooks.

And as another commenter suggested, use Google Docs. These should be super easy to edit with zero friction. The requirement of creating a PR increases the odds that something will get skipped.

FWIW, several of my previous teams required more than five nines of uptime as we'd lose millions for outages. We had a very mature process.

The book is overlong for what it has to say, and stuffed with things that are not really checklists (architect's building plans).

Fortunately there is a better version with not just all the meat, but also the best anecdotes included: Gawande's original article in The New Yorker.

https://www.newyorker.com/magazine/2007/12/10/the-checklist

Runbooks are, in my experience at least, generally sticking plasters over systems that are convoluted, unreliable, or otherwise broken. They're a symptom of teams that are running so hot that they can only put out fires, not prevent them. Again, just my experience, but whenever I see an extensive list of runbooks I run a mile.

What is the difference (if any) between a runbook and a process? Genuine question; I am interested to hear people's answers.

Runbooks are very common in safety critical environments. E.g. aviation is completely driven by checklists. Everything has a checklist. Even amateur pilots tend to stick to their checklists.

There are a few useful properties with checklists:

- They stop you from having to improvise actions when time is too short for that. People under stress take bad decisions. It's better for them to stick to a pre-defined plan.

- They stop you from making preventable mistakes. People forget stuff. Even trained people. Especially routine stuff.

- They ensure some level of uniformity to people's actions. So, they can check each other's work and do some cross-checking.

For IT operations, run books are great. Especially if you can automate them. CI/CD is basically an automated run book. It simplifies the decision process for humans: do I want to put this live: yes/no?

But what do you do when the database goes down or there is some major hardware failure and you have to restore from a backup? In a lot of organizations, this is still a major crisis. There might be a run book but chances are nobody has practiced this in production in ages. Thinking through ahead of time what you would do is more than half of the success. It doesn't matter if it involves some manual steps as long as you know what they are. In aviation, a lot of stuff is not automated at all. And they still use run books and checklists. These are a great substitute for automation. Newer planes self-check a lot of stuff that you used to have to do manually.

Ideally you have both automation and run books. And automation can fail of course. What do you do when your cloudformation stack is corrupted or when your terraform is out of sync with what remains of your production environment? It happens and it's a bad time for experimentation. Usually such events lead to runbook updates. You learn some new stuff about exciting failure modes you did not consider before.

Having the practice of documenting non obvious things is what separates a good from an excellent engineer. I actually do it to help my future self as well. Because I know I'll go down a rabbit hole of reinventing things I already figured out a few months down the line.