21–30 of 37 posts
This is awesome, i've seen so many static runbooks (like confluence) and SREs will scan it once, not find what they need and then go wake up a senior dev. Pre-programmed scripts could go a long way in giving the SRE the ability to go that extra step, which could be vital to solving the problem faster.
This is a great idea! But I feel better served by an existing workflow tool, such as Airflow?
Great job nonetheless!
It would be so cool to also have access to GCP resources! Great job nonetheless!
But nowadays I think if I can automate a runbook can I not just make the system heal itself automatically? If you have repeated problems with known solutions you should invest in toil reduction to stop having those repeated problems.
What am I missing? I think I must be missing something because these kinds of things keep popping up.
Whenever I see tools like this I always think "that wouldve been great at my old job where we didn't do post mortems" But nowadays I think if I can automate a runbook can I not just make the system heal itself automatically? If you have repeated problems with known solutions you should invest in toil reduction to stop having those repeated problems. What am I missing? I think I must be missing something because these…
The runbooks are still codified by a human in the current scenario. We are experimenting with some data to see if we can generate accurate runbooks for different scenarios but haven't found much luck with it yet. I do think that some % of issues will be abstracted in near future with machines doing the healing automatically.
> you should invest in toil reduction to stop having those repeated problems.
Most teams I speak to say that they try their best to avoid repeating the same issue again. Users typically use PlayBooks for:
(a) A generic scenario where you have an issue reported / alerted and you are testing 3-4 hypotheses / potential failure reasons at once.
(b) You want to run some definitive sequence of steps.
- The “rename step” functionality is not intuitive. I expecting tapping on the step name to “unfold” the step and show me the full details, not start the renaming process. After tapping it, I still didn't realize what was happening; i thought perhaps it had executed the step, which the check mark indicating completion or success. It wasn’t clear that it was an input box since it didn’t have focus, and it wasn’t clear that the check mark was a button.
I would have guessed that the pencil icon perhaps was the rename action, though it still did not put focus on the input box. There shouldn't be a second step needed to focus the input box.
- It’s not clear what defines the “type” of each step; eg whether it’s a log filter, or dh query, or shell command, etc. It seems like it’s the “Data” field, although the name doesn’t make much sense. The field does not seem to be editable; I would have expected it to be a dropdown list with other possible step types listed. If it is intended not to be changeable, then it probably shouldn’t be an input element. There’s a “reload”(?) icon next to it, but I have no idea what that does.
Feedback on the sample playbook: - The “rename step” functionality is not intuitive. I expecting tapping on the step name to “unfold” the step and show me the full details, not start the renaming process. After tapping it, I still didn't realize what was happening; i thought perhaps it had executed the step, which the check mark indicating completion or success. It wasn’t clear that it was an input box since it didn’…
We deployed the change to make it intuitive (similar to what is suggested) yesterday. It's still in integration branch so awaiting merging in main on this.
> i thought perhaps it had executed the step, which the check mark indicating completion or success.
Noted.
> It wasn’t clear that it was an input box since it didn’t have focus, and it wasn’t clear that the check mark was a button.
Noted.
> It seems like it’s the “Data” field, although the name doesn’t make much sense.
It is indeed a dropdown list but we had hard-coded it for sandbox so user can't change the source of an existing step. It is changeable when you host your own version or when you add a new step in sandbox.
> There’s a “reload”(?) icon next to it, but I have no idea what that does.
In case user decides to add a new source on-the-go (say in another tab), reload helps fetch the same list again.
Overall, I do understand that some parts of it are unintuitive and is a focus area for us to improve it asap.