Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On-call problems – here are mine. Do you feel the same way?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 32 posts

On-call problems – here are mine. Do you feel the same way?

#1

Post by on-call_guy »

Hi Folks,

Being on-call has been one of the most painful part of my job as a software engineer now a days. There were a lot of stressful weeks I had spend with completely demotivated about how much time I have been spending on these issues which can be spent on the innovation. So I have listed my top issues in below ranks. I was wondering if others feel the same pain? I also wonder why can there be a solution built for these?

1. I do not have enough information in alert to jump right on the resolution

2. It’s not easy to find similar alerts triggered recently so that I can go back and find how they were fixed?

3. I don’t find runbooks useful most of the times as they are not up to date

4. I don’t know if there were any recently merged changes which caused these alerts/incidents

5. A lot of the time, I don’t know whom to reach out to if this alert is from other team.

6. I have to go to multiple systems to update the statuses or notes

7. I have to summarize all the details again as a part of on-call handoff summary doc at the end of the rotation

Re: On-call problems – here are mine. Do you feel the same way?

#2

Post by GianFabien »

I worked for a systems integration and management firm for five years. We avoided the sorts of problems you describe by having a management who gave us the best tools and training for our work and in return demanded exacting documentation which had to be kept up to date as part of our work. Logs and alerts were refined to eliminate confusion. We were tasked with implementing scripts to correct, mitigate the effect of problems.

Being on-call is a challenge, but also an opportunity to improve processes. Your management should empower the team to fix the processes.

Re: On-call problems – here are mine. Do you feel the same way?

#3

Post by on-call_guy »

GianFabien wrote: post #2

I worked for a systems integration and management firm for five years. We avoided the sorts of problems you describe by having a management who gave us the best tools and training for our work and in return demanded exacting documentation which had to be kept up to date as part of our work. Logs and alerts were refined to eliminate confusion. We were tasked with implementing scripts to correct, mitigate the effect of…

Thank you. Its a great point and totally agree that a good management plays a big role in making life little easy. We did raise it to our management. But one of the limitations from their end is as well too many different tools and scattered information which do not give them full insights. For examples, its very hard to know which runbooks are stale and needs updates unless your frequently review them. Curious to know how such problems were solved?

Re: On-call problems – here are mine. Do you feel the same way?

#4

Post by dudul »

on-call_guy wrote: post #3
GianFabien wrote: post #2

I worked for a systems integration and management firm for five years. We avoided the sorts of problems you describe by having a management who gave us the best tools and training for our work and in return demanded exacting documentation which had to be kept up to date as part of our work. Logs and alerts were refined to eliminate confusion. We were tasked with implementing scripts to correct, mitigate the effect of…

Thank you. Its a great point and totally agree that a good management plays a big role in making life little easy. We did raise it to our management. But one of the limitations from their end is as well too many different tools and scattered information which do not give them full insights. For examples, its very hard to know which runbooks are stale and needs updates unless your frequently review them. Curious to kn…

Why don't you update the run books? Why don't you modify the alerts/logs to give you more information? Why don't you create the missing run books when you run into undocumented issues?

Re: On-call problems – here are mine. Do you feel the same way?

#5

Post by hanifak »

I’m actually building exactly this. It’s a simple on-call and incident management platform that covers many of your frustrations since I’ve had the same ones. I’d love to talk to you about it and get your feedback on my progress so far if you’re interested.

Re: On-call problems – here are mine. Do you feel the same way?

#6

Post by db48x »

dudul wrote: post #4
on-call_guy wrote: post #3

Earlier quoted context omitted.

Thank you. Its a great point and totally agree that a good management plays a big role in making life little easy. We did raise it to our management. But one of the limitations from their end is as well too many different tools and scattered information which do not give them full insights. For examples, its very hard to know which runbooks are stale and needs updates unless your frequently review them. Curious to kn…

Why don't you update the run books? Why don't you modify the alerts/logs to give you more information? Why don't you create the missing run books when you run into undocumented issues?

It sounds like none of those things have an owner who is tasked with keeping them up to date and correct. All the work that needs to be done needs to have a specific well—documented owner, otherwise diffusion of responsibility ensures that it will eventually fall through the cracks.

Re: On-call problems – here are mine. Do you feel the same way?

#7

Post by rozenmd »

on-call_guy wrote: post #3
GianFabien wrote: post #2

I worked for a systems integration and management firm for five years. We avoided the sorts of problems you describe by having a management who gave us the best tools and training for our work and in return demanded exacting documentation which had to be kept up to date as part of our work. Logs and alerts were refined to eliminate confusion. We were tasked with implementing scripts to correct, mitigate the effect of…

Thank you. Its a great point and totally agree that a good management plays a big role in making life little easy. We did raise it to our management. But one of the limitations from their end is as well too many different tools and scattered information which do not give them full insights. For examples, its very hard to know which runbooks are stale and needs updates unless your frequently review them. Curious to kn…

It sounds like your team lacks a culture of continuous improvement - IMO in a product team on-call's full-time job is to make the next on-call engineer's job easier through deleting irrelevant alerts, automating fixes, and generally making the system more stable.

I wrote a longer guide about this here: https://onlineornot.com/incident-management/on-call/improvin...

Re: On-call problems – here are mine. Do you feel the same way?

#9

Post by GianFabien »

db48x wrote: post #6
dudul wrote: post #4

Earlier quoted context omitted.

Why don't you update the run books? Why don't you modify the alerts/logs to give you more information? Why don't you create the missing run books when you run into undocumented issues?

It sounds like none of those things have an owner who is tasked with keeping them up to date and correct. All the work that needs to be done needs to have a specific well—documented owner, otherwise diffusion of responsibility ensures that it will eventually fall through the cracks.

Management's job is be "the owner". They are ultimately responsible to make sure that there is no diffusion of responsibility.

In our weekly meetings, recurring problems were identified and fixes implemented. No call was considered completed and closed until all relevant documents had been updated as appropriate. At the yearly review the quality of your documentation was as important as your time to respond, time to fix. That is how mission critical on-call work should be handled.

Re: On-call problems – here are mine. Do you feel the same way?

#10

Post by on-call_guy »

rozenmd wrote: post #7
on-call_guy wrote: post #3

Earlier quoted context omitted.

Thank you. Its a great point and totally agree that a good management plays a big role in making life little easy. We did raise it to our management. But one of the limitations from their end is as well too many different tools and scattered information which do not give them full insights. For examples, its very hard to know which runbooks are stale and needs updates unless your frequently review them. Curious to kn…

It sounds like your team lacks a culture of continuous improvement - IMO in a product team on-call's full-time job is to make the next on-call engineer's job easier through deleting irrelevant alerts, automating fixes, and generally making the system more stable. I wrote a longer guide about this here: https://onlineornot.com/incident-management/on-call/improvin...

Yeah, I must agree it is a cultural issue at some extent. But honestly the on-call my current company is quite demanding. So during the on-call week, though engineers try to improve it they always run out of the time or miss few things which then puts burden on future on-call.

I think there should be a nice light weight tool which should give a clear summary and tracking mechanism which make this a quicker tasks. Even just to tag the runbooks which are not updated. All those notes get lost in documentations and never referred back.

Post reply on HN