Earlier quoted context omitted.
It sounds like none of those things have an owner who is tasked with keeping them up to date and correct. All the work that needs to be done needs to have a specific well—documented owner, otherwise diffusion of responsibility ensures that it will eventually fall through the cracks.
Management's job is be "the owner". They are ultimately responsible to make sure that there is no diffusion of responsibility. In our weekly meetings, recurring problems were identified and fixes implemented. No call was considered completed and closed until all relevant documents had been updated as appropriate. At the yearly review the quality of your documentation was as important as your time to respond, time to…
On-call problems – here are mine. Do you feel the same way?
11–20 of 32 posts
Re: On-call problems – here are mine. Do you feel the same way?
#12e.g instead of complaining about outdated runbooks, you can just update them.
Re: On-call problems – here are mine. Do you feel the same way?
#13I’m actually building exactly this. It’s a simple on-call and incident management platform that covers many of your frustrations since I’ve had the same ones. I’d love to talk to you about it and get your feedback on my progress so far if you’re interested.
Even xMatter which my wife's team use but never login as it crashes frequently and prefer to debug through logs :P
Re: On-call problems – here are mine. Do you feel the same way?
#14You can improve all of these. Why aren’t you? e.g instead of complaining about outdated runbooks, you can just update them.
Again, I am talking about the teams who are heavily loaded with alerts and incidences during on-call. In general, all these pain points very much vary depending on the on-call load. We also have a teams who do manage all of these easily as their on-call load is quite manageable. But solving for them in one place so that everyone from the team is on the same would be amazing.
Re: On-call problems – here are mine. Do you feel the same way?
#15I think the biggest hussle during on-call is that a lot of stuffs have no clear ownership so I don't know who to throw the hot potato to. We are switching to a better solution with clear ownership so hopefully it helps.
Re: On-call problems – here are mine. Do you feel the same way?
#16On call monitoring responsibilities for a certain time period should be separate from resolution duties.
In other words, aside for some well defined ops issues that have clear runbooks, the role of the person monitoring should be find out or know who to escalate to, not resolve.
It's actually a great onboarding activity as it exposes new staff members to parts of the infra and operations that their managers/peers might have neglected to mention.
The second way to alleviate the issues is to pair a person such as yourself with a person that has a lot of institutional knowledge so that you can triage together, learn from them, and update the docs so the organisation as a whole has better resources. Eventually, the percentage of incidents where you don't have the institutional knowledge to know how to proceed will decrease to the point where it's mostly safe for you to do on-call on your own.
Then eventually you become the experienced on-call person that gets paired with the new employees that are gaining that institutional knowledge.
Re: On-call problems – here are mine. Do you feel the same way?
#17I think the biggest hussle during on-call is that a lot of stuffs have no clear ownership so I don't know who to throw the hot potato to. We are switching to a better solution with clear ownership so hopefully it helps.
lol...yeah we have that established within the team now. But still a challenge to find out a right contact across teams at 3am in the morning if the issue is from another team. May be we should build a service level ownership list so that we can tag them (in addition to their on-call). Curious to know what level of ownership you were referring about?
Re: On-call problems – here are mine. Do you feel the same way?
#18I’m actually building exactly this. It’s a simple on-call and incident management platform that covers many of your frustrations since I’ve had the same ones. I’d love to talk to you about it and get your feedback on my progress so far if you’re interested.
Ok...Would love to know more about what is out there. The problem with current alerting solutions like PagerDuty, they are every extensive in terms of what they offer (scheduling, reliability, etc) but not quite tailored towards needs of on-call engineer to easily tag something or for management to get a view which alerts/incidents need attention. Even xMatter which my wife's team use but never login as it crashes fr…
I totally agree that the tools which exist today cater towards the buyer. That is, the people with purchasing power who typically aren’t on-call. I’m building with a focus on the on-call experience for the people who are actually on call.
Re: On-call problems – here are mine. Do you feel the same way?
#19Earlier quoted context omitted.
It sounds like none of those things have an owner who is tasked with keeping them up to date and correct. All the work that needs to be done needs to have a specific well—documented owner, otherwise diffusion of responsibility ensures that it will eventually fall through the cracks.
Management's job is be "the owner". They are ultimately responsible to make sure that there is no diffusion of responsibility. In our weekly meetings, recurring problems were identified and fixes implemented. No call was considered completed and closed until all relevant documents had been updated as appropriate. At the yearly review the quality of your documentation was as important as your time to respond, time to…
Re: On-call problems – here are mine. Do you feel the same way?
#20I’m actually building exactly this. It’s a simple on-call and incident management platform that covers many of your frustrations since I’ve had the same ones. I’d love to talk to you about it and get your feedback on my progress so far if you’re interested.
Ok...Would love to know more about what is out there. The problem with current alerting solutions like PagerDuty, they are every extensive in terms of what they offer (scheduling, reliability, etc) but not quite tailored towards needs of on-call engineer to easily tag something or for management to get a view which alerts/incidents need attention. Even xMatter which my wife's team use but never login as it crashes fr…