Keep calm and use the runbook

In the ideal world, yes.

Some systems are, for any number of reasons, strongly resistant to automation. The obvious case is where security policies may require a human in the loop. Then there are the Rube Goldberg systems where the interface to the system involves VNC to a MacOS machine that runs a Virtualbox with a Linux guest where a native application modifies some database you have no other access to. Etc.

I think my new favourite way of managing runbooks is to actually build them into a file tree of a bunch of simple python subcommand scripts, and have a run.sh script that scans the file system and uses argparse to construct a cli to call each script.

  # call ./runbooks/stack/update_secret.py
  # could update a secret in a vault, or update it in your deployed app
  ./run.sh stack update_secret --env=dev --name=foo --file=secret.txt

Most of the time my python scripts are glorified CLI commands like `docker service update` that are called through subprocess, so you shouldn't need to install dependencies beyond what you'd be typing in the CLI. It's also easy to add a verbose option to print out the commands it runs so you can do it manually.

  # call ./runbooks/services/build.py
  ./run.sh services build -v
  > #--- Building images ---
  > #> DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --label "myapp" -t example-admin-ui:local "./admin-ui"
  > #> DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --label "myapp" -t example-frontend:local "./front-end"
  > #> DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --label "myapp" -t example-nginx:local "./nginx"

Anything that can't be automated prints out an input line that gives instructions on what to do and just waits for you to input "yes/no"

  # call ./runbooks/get_crash_report.py
  ./run.sh get_crash_report --out=./crashes/
  > # Copying crashes from AWS to './crashes/
  > # Manual Step: Fill out crashes spreadsheet: docs.google/example_sheet
  > Continue [y/n]? 
  

The other really nice thing with this setup is the run.sh script is able to build up --help commands that can print out what actions are available and what params they use cause it's just python argparse. Makes discovery of what to do or looking up params really quick.

At this point, the only culture you need to build is one where everyone's supposed to use the run.sh scripts and not do things manually. This enforces people to fix the scripts when something changes.

YMMV, but I've found this has simplified a lot of processes for myself at least.

Water flows down the easiest path. Make runbooks (and their maintenance) easy. This requires management buy-in to create incentives that make runbooks the easiest path.

In my experience, getting management on-board is the most difficult part since engineers will be lazy (I mean this here in a positive, efficient way) and do the easier thing.

I’ve set up runbooks on a few different teams at different companies and I’ve found some strategies that help. Feel free to pick and choose what would make sense from this list:

- Give runbooks (and on-call) time. Subtract the number of expected on-call weeks from your quarterly plan. This is where management buy-in is the most important since on the surface it impacts “throughput” or whatever metric is in vogue. As an aside, it’s unfortunate that most managers (IME) don’t track morale or operational burden as metrics…

- When an engineer is on-call they can do housekeeping, sweep runbooks, dashboards, etc. You want the on-call engineer to have time to do RCAs, and make reasonable changes in response to incidents. Downtime can be spent cleaning and updating. We’ve called out “operational improvements” during standup.

- Every alert links to a runbook. No alerts can be merged without a runbook. Dashboard, etc, are linked from the runbook. The engineer that created the system/alert might not be the first to encounter it at 3am and the person on-call should know what to do.

- Every time you get an alert, scan the runbook. Even if you are a system expert. Something may have changed or someone may have modified a process to be more automated. You could be pleasantly surprised that an alert is now easier to manage due to someone else’s operational improvements that you forgot about.

- If a runbook is wrong or out of date, fix it right then or flag it as a problem and bring it up during standup. The runbook is a part of the software “package” and issues should be treated like bugs that should be addressed.

- Do an on-call review and track the on-call engineer’s sentiment along with metrics around how many wake-ups, weekend alerts, etc. up-to-date, clear runbooks make on-call much less frustrating for systems the on all engineer didn’t directly write. It’s also a good forum for a team member feels they are taking on more of the operational upkeep.

- Call out and celebrate excellent, up to date runbooks! If you were on call and a teammate’s runbook was great and let you effectively manage an incident in a part of the system you don’t have as much experience in, let them know!

Runbook development/maintenance is a part of software development/maintenance (IMO). If someone isn’t pulling their weight with runbooks, then they aren’t doing all of their job. I see it the same as someone who only wants to work on greenfield projects. There’s some percentage of shit you need to deal with as a software engineer, operations is one of them. At least for online systems.

Interesting experience, thanks.

It's not really a call to predict anything.

If you're hiring some who you're going to need to mop the store, and during the interview you make it clear that they're going to need to mop the store, and you ask them if they will mop the store, and it turns out that they don't mop the store, that's considered a sufficient reason to get rid of them.

Having an explicit discussion on the topic of "Will you do this thing that we expect you to do?" means that (a) the interview tracks what actual day-to-day expectations are, and (b) if they don't do the thing, there's no need to feel awkward about wanting to not keep them around after promising something and failing to follow through on it.