Live data from Hacker News

On-call problems – here are mine. Do you feel the same way?

news.ycombinator.com

11–20 of 32 posts

Re: On-call problems – here are mine. Do you feel the same way?

#11
post #6

Earlier quoted context omitted.

It sounds like none of those things have an owner who is tasked with keeping them up to date and correct. All the work that needs to be done needs to have a specific well—documented owner, otherwise diffusion of responsibility ensures that it will eventually fall through the cracks.

Management's job is be "the owner". They are ultimately responsible to make sure that there is no diffusion of responsibility. In our weekly meetings, recurring problems were identified and fixes implemented. No call was considered completed and closed until all relevant documents had been updated as appropriate. At the yearly review the quality of your documentation was as important as your time to respond, time to…

That's great. I think one of the issue in our process is we use wiki for on-call summary/hand-off notes. That's not ALWAYS very helpful as it has a dependency what engineers add to them. Also time and severity of the alerts make a difference as well. E.g. if they are triggered in the night/unfriendly time the first intuition of the engineer is to fix it and not to make a note or document unless there a easy way to do so. We use PagerDuty and I dont think it provides easy way to make those note or comments. So that leaves it to the engineers who need to do it after the fact. Some teammates do it rigorously where some dont. I think Management's challenge is also they can only push so much as it becomes an attrition risk :(

Re: On-call problems – here are mine. Do you feel the same way?

#13
post #5

I’m actually building exactly this. It’s a simple on-call and incident management platform that covers many of your frustrations since I’ve had the same ones. I’d love to talk to you about it and get your feedback on my progress so far if you’re interested.

Ok...Would love to know more about what is out there. The problem with current alerting solutions like PagerDuty, they are every extensive in terms of what they offer (scheduling, reliability, etc) but not quite tailored towards needs of on-call engineer to easily tag something or for management to get a view which alerts/incidents need attention.

Even xMatter which my wife's team use but never login as it crashes frequently and prefer to debug through logs :P

Re: On-call problems – here are mine. Do you feel the same way?

#14

You can improve all of these. Why aren’t you? e.g instead of complaining about outdated runbooks, you can just update them.

I agree they can be improved. But they are not one-time activity. It should be a continuous process in order to really be efficient. Also every on-call person needs to be diligent about it. Otherwise, it aggregates in future.

Again, I am talking about the teams who are heavily loaded with alerts and incidences during on-call. In general, all these pain points very much vary depending on the on-call load. We also have a teams who do manage all of these easily as their on-call load is quite manageable. But solving for them in one place so that everyone from the team is on the same would be amazing.

Re: On-call problems – here are mine. Do you feel the same way?

#15

I think the biggest hussle during on-call is that a lot of stuffs have no clear ownership so I don't know who to throw the hot potato to. We are switching to a better solution with clear ownership so hopefully it helps.

lol...yeah we have that established within the team now. But still a challenge to find out a right contact across teams at 3am in the morning if the issue is from another team. May be we should build a service level ownership list so that we can tag them (in addition to their on-call). Curious to know what level of ownership you were referring about?

Re: On-call problems – here are mine. Do you feel the same way?

#16
Some of these need to be fixed at a higher level.

On call monitoring responsibilities for a certain time period should be separate from resolution duties.

In other words, aside for some well defined ops issues that have clear runbooks, the role of the person monitoring should be find out or know who to escalate to, not resolve.

It's actually a great onboarding activity as it exposes new staff members to parts of the infra and operations that their managers/peers might have neglected to mention.

The second way to alleviate the issues is to pair a person such as yourself with a person that has a lot of institutional knowledge so that you can triage together, learn from them, and update the docs so the organisation as a whole has better resources. Eventually, the percentage of incidents where you don't have the institutional knowledge to know how to proceed will decrease to the point where it's mostly safe for you to do on-call on your own.

Then eventually you become the experienced on-call person that gets paired with the new employees that are gaining that institutional knowledge.

Re: On-call problems – here are mine. Do you feel the same way?

#17

I think the biggest hussle during on-call is that a lot of stuffs have no clear ownership so I don't know who to throw the hot potato to. We are switching to a better solution with clear ownership so hopefully it helps.

lol...yeah we have that established within the team now. But still a challenge to find out a right contact across teams at 3am in the morning if the issue is from another team. May be we should build a service level ownership list so that we can tag them (in addition to their on-call). Curious to know what level of ownership you were referring about?

Basically the same as yours: who to call when shtf. The tricky part is that managers don't do trench work so once the developer responsible does not reply then the oncall has to figure it out.

Re: On-call problems – here are mine. Do you feel the same way?

#18
post #5

I’m actually building exactly this. It’s a simple on-call and incident management platform that covers many of your frustrations since I’ve had the same ones. I’d love to talk to you about it and get your feedback on my progress so far if you’re interested.

Ok...Would love to know more about what is out there. The problem with current alerting solutions like PagerDuty, they are every extensive in terms of what they offer (scheduling, reliability, etc) but not quite tailored towards needs of on-call engineer to easily tag something or for management to get a view which alerts/incidents need attention. Even xMatter which my wife's team use but never login as it crashes fr…

Do you have an email I could reach you at? I’d like to share what I’m thinking the solution looks like and hear more about the challenges you’ve faced.

I totally agree that the tools which exist today cater towards the buyer. That is, the people with purchasing power who typically aren’t on-call. I’m building with a focus on the on-call experience for the people who are actually on call.

Re: On-call problems – here are mine. Do you feel the same way?

#19
post #6

Earlier quoted context omitted.

It sounds like none of those things have an owner who is tasked with keeping them up to date and correct. All the work that needs to be done needs to have a specific well—documented owner, otherwise diffusion of responsibility ensures that it will eventually fall through the cracks.

Management's job is be "the owner". They are ultimately responsible to make sure that there is no diffusion of responsibility. In our weekly meetings, recurring problems were identified and fixes implemented. No call was considered completed and closed until all relevant documents had been updated as appropriate. At the yearly review the quality of your documentation was as important as your time to respond, time to…

Yea, that’s not uncommon. Personally I prefer to give each document a specific owner, but either way you do it someone has to be tasked with ensuring that the documentation is correct.

Re: On-call problems – here are mine. Do you feel the same way?

#20
post #5

I’m actually building exactly this. It’s a simple on-call and incident management platform that covers many of your frustrations since I’ve had the same ones. I’d love to talk to you about it and get your feedback on my progress so far if you’re interested.

Ok...Would love to know more about what is out there. The problem with current alerting solutions like PagerDuty, they are every extensive in terms of what they offer (scheduling, reliability, etc) but not quite tailored towards needs of on-call engineer to easily tag something or for management to get a view which alerts/incidents need attention. Even xMatter which my wife's team use but never login as it crashes fr…

I just stood up my landing page if you want to keep up to date: https://simpleoncall.com/.
Post reply on HN