Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On-call problems – here are mine. Do you feel the same way?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: On-call problems – here are mine. Do you feel the same way?

#21

Post by rozenmd »

on-call_guy wrote: post #10
rozenmd wrote: post #7

Earlier quoted context omitted.

It sounds like your team lacks a culture of continuous improvement - IMO in a product team on-call's full-time job is to make the next on-call engineer's job easier through deleting irrelevant alerts, automating fixes, and generally making the system more stable. I wrote a longer guide about this here: https://onlineornot.com/incident-management/on-call/improvin...

Yeah, I must agree it is a cultural issue at some extent. But honestly the on-call my current company is quite demanding. So during the on-call week, though engineers try to improve it they always run out of the time or miss few things which then puts burden on future on-call. I think there should be a nice light weight tool which should give a clear summary and tracking mechanism which make this a quicker tasks. Eve…

In previous teams, we just used a JIRA backlog to manage these tasks

Re: On-call problems – here are mine. Do you feel the same way?

#22

Post by on-call_guy »

hanifak wrote: post #20
on-call_guy wrote: post #13

Earlier quoted context omitted.

Ok...Would love to know more about what is out there. The problem with current alerting solutions like PagerDuty, they are every extensive in terms of what they offer (scheduling, reliability, etc) but not quite tailored towards needs of on-call engineer to easily tag something or for management to get a view which alerts/incidents need attention. Even xMatter which my wife's team use but never login as it crashes fr…

I just stood up my landing page if you want to keep up to date: https://simpleoncall.com/ .

Thanks for the link. It doesn't have much details. I have signed up though. Looking forward for getting more details about the solutions.

Re: On-call problems – here are mine. Do you feel the same way?

#25

Post by anon743448 »

on-call_guy wrote: post #15
markus_zhang wrote: post #8

I think the biggest hussle during on-call is that a lot of stuffs have no clear ownership so I don't know who to throw the hot potato to. We are switching to a better solution with clear ownership so hopefully it helps.

lol...yeah we have that established within the team now. But still a challenge to find out a right contact across teams at 3am in the morning if the issue is from another team. May be we should build a service level ownership list so that we can tag them (in addition to their on-call). Curious to know what level of ownership you were referring about?

[dead]

Re: On-call problems – here are mine. Do you feel the same way?

#26

Post by mirage_18 »

I previously worked at a popular startup. I was part of a team that owned many business critical services. Our on-call was brutal. I have firsthand experienced most of the problems that you are talking about.

But interestingly, we solved these problem back then using an internal tool. Here is how the internal tool solved the problems -

1. It had integration with all internal tools like task management, alert management, monitoring systems and pagerduty. It offered 1 central dashboard where it all came together

2. Each team in pagerduty can see details of all alerts that happened in given shift/rotation. So anyone can go there anytime to see what alerts are fired, when and to whom.

3. Each alert had option to mark with various tags like noisy, non-actionable, etc. Additionally a note and follow up task links can be added.

The tool solves some of the problems you mentioned properly. e.g. You don't need to write a summary document. It all gets captured there and can be easily viewed in the handoff meetings. With tags, you can easily find bad alerts or alerts with outdated runbooks. Its easy to hold team-members accountable if they are not following process/best practices.

Oncall is a heavy process but IMO with right tooling, a lot of problems can be solved properly.

PS - I didn't create the tool but I used it extensively to get my team's oncall under control.

Re: On-call problems – here are mine. Do you feel the same way?

#27

Post by on-call_guy »

rozenmd wrote: post #21
on-call_guy wrote: post #10

Earlier quoted context omitted.

Yeah, I must agree it is a cultural issue at some extent. But honestly the on-call my current company is quite demanding. So during the on-call week, though engineers try to improve it they always run out of the time or miss few things which then puts burden on future on-call. I think there should be a nice light weight tool which should give a clear summary and tracking mechanism which make this a quicker tasks. Eve…

In previous teams, we just used a JIRA backlog to manage these tasks

Yeah, JIRA could be handy and useful though you need to create tickets for every tasks with a rigorous monitoring with other backlog and story items.

Re: On-call problems – here are mine. Do you feel the same way?

#28

Post by on-call_guy »

markus_zhang wrote: post #17
on-call_guy wrote: post #15

Earlier quoted context omitted.

lol...yeah we have that established within the team now. But still a challenge to find out a right contact across teams at 3am in the morning if the issue is from another team. May be we should build a service level ownership list so that we can tag them (in addition to their on-call). Curious to know what level of ownership you were referring about?

Basically the same as yours: who to call when shtf. The tricky part is that managers don't do trench work so once the developer responsible does not reply then the oncall has to figure it out.

Got it. I wish there was an up to date service level owner identified which will then reduce this to just a lookup and tagging. I have seen few engineering teams in other companies started doing that.

Re: On-call problems – here are mine. Do you feel the same way?

#29

Post by on-call_guy »

mirage_18 wrote: post #26

I previously worked at a popular startup. I was part of a team that owned many business critical services. Our on-call was brutal. I have firsthand experienced most of the problems that you are talking about. But interestingly, we solved these problem back then using an internal tool. Here is how the internal tool solved the problems - 1. It had integration with all internal tools like task management, alert manageme…

I see...how does it really work though? Like if I am on-call for this week- would it show all my alerts in one place and then allow me to take some kind of actions? How does it solve the other issues like stale runbooks, etc?

Re: On-call problems – here are mine. Do you feel the same way?

#30

Post by on-call_guy »

ac2u wrote: post #16

Some of these need to be fixed at a higher level. On call monitoring responsibilities for a certain time period should be separate from resolution duties. In other words, aside for some well defined ops issues that have clear runbooks, the role of the person monitoring should be find out or know who to escalate to, not resolve. It's actually a great onboarding activity as it exposes new staff members to parts of the…

Yep, we do this in our team too (day time only though) and call it as "on-call buddy" for first couple of rotations.
Post reply on HN