31–40 of 51 posts
What happens if you are in a dead zone and you need to look at the owner's manual in an emergency?
Too bad.
Earlier quoted context omitted.
Can you explain more about this? Most of the runbooks at my current job exist because it was too painful to leave them unwritten, and get used frequently.
Runbooks are, in my experience at least, generally sticking plasters over systems that are convoluted, unreliable, or otherwise broken. They're a symptom of teams that are running so hot that they can only put out fires, not prevent them. Again, just my experience, but whenever I see an extensive list of runbooks I run a mile.
So pretty much every system not written from scratch in the past five years?
Earlier quoted context omitted.
Runbooks are, in my experience at least, generally sticking plasters over systems that are convoluted, unreliable, or otherwise broken. They're a symptom of teams that are running so hot that they can only put out fires, not prevent them. Again, just my experience, but whenever I see an extensive list of runbooks I run a mile.
generally sticking plasters over systems that are convoluted, unreliable, or otherwise broken So pretty much every system not written from scratch in the past five years?
It is obvious to me that keeping the runbook up to date is really important. But how do you foster a culture in which everyone keeps it up to date? Many people don't even read the readme, let alone correct it. Is it cultural? If you are so privileged that your books are updated, please share with us your ways!
Round robin on-call
Putting people on an on-call rota gives them lots more experiences and takes them out of their silos and allows them to see the effect that their work has on everybody else. It also gives them the time to be updating docs without feeling guilty about not coding.
Earlier quoted context omitted.
Round robin on-call
This is the best suggestion imho. Most developers simply don't enjoy documentation because it doesn't "do anything" and good luck to those who say "employ people who will do it" because we all want to do that but when people don't do it, and you fire them, you have an even bigger issue with morale and rehiring. Putting people on an on-call rota gives them lots more experiences and takes them out of their silos and al…
It is obvious to me that keeping the runbook up to date is really important. But how do you foster a culture in which everyone keeps it up to date? Many people don't even read the readme, let alone correct it. Is it cultural? If you are so privileged that your books are updated, please share with us your ways!
It reminded me of a simple but not so obvious concept - do nothing scripts. https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
It is obvious to me that keeping the runbook up to date is really important. But how do you foster a culture in which everyone keeps it up to date? Many people don't even read the readme, let alone correct it. Is it cultural? If you are so privileged that your books are updated, please share with us your ways!
They were very reliant on runbooks for operational things.
Problem was, at least half of them were out of date.
Need to access logs on a machine when they aren't being sent someplace like {Datadog, Cloudwatch, etc.}? Here are the SSH instructions. But wait, this key is no longer in the secure vault. etc. etc. with no updated information on how to get the new IAM or whatever permissions needed for SSH access.
I think calling it a culture problem is a cop-out, unfortunately.
Documentation woes, in and of themselves, exist everywhere in this industry. I don't know what the solution is, but I just don't think it's a culture thing.
If anything, it should be a mental health thing. You're on-call and get paged at 3AM about some metric drastically falling and you can't get the logs on your usual log system and need to log into a machine or cluster directly. And since your team is heavily runbook-reactive (new term I just coined to describe a team reliant on runbooks to solve "known" problems rather than learned knowledge), you're totally stuck and under a lot of pressure precisely because the runbook is no longer applicable. This is never fun for anyone.
One possible approach is setting up a type of Red vs Blue team monthly or quarterly event. Half the team spends the week randomly "breaking" things (ideally in a stage or other, more isolated environment) and the other half gets the pages, learns what runbooks are out of date or otherwise no longer correct, fix the issue both operationally and in the runbooks. Additionally, in the process one team learns about what kinds of issues can break the system(s) and the other team learns about what kinds of fixes can return the system(s) to working order.
Then switch off roles every other time.
A European car company--that I can't name--proposes to take the owner's manual out of the glovebox, and thanks to the Internet of things--put it online. Their cars are exported to countries where there are Internet dead zones. What happens if you are in a dead zone and you need to look at the owner's manual in an emergency? Too bad.