Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being on call sucks

bobbiechen.com

Post reply on HN
Print view View original

71–80 of 282 posts

Re: Being on call sucks

#71

Post by volume »

francis-io wrote: post #18
tayo42 wrote: post #12

Yeah oncall is horrible. If things needed to be up 24/7, then some team should be staffed 24/7 around the world. The worse part of oncall is the control of your life it has. for one week I can't do anything I would normally do. (if your company actually compensates for this, let me know where i can a apply, or better, if it doesn't have oncall at all!) Of course managers are never oncall 24/7. The worse is they give…

The problem with oncall teams that are different to the usual SRE/DevOps teams are the lack of understanding of the system. This can obviously be fixed with good documentation, but in reality, no one has good enough docs. The second problem is actually building that team with the skill needed. Someone with the skill to fix complex systems is not going to want to stick around as 1st line on call support.

Most places at some scale have some meaningfully defined escalation path. That way you staff people with varying understanding of the system(s).

> This can obviously be fixed with good documentation, but in reality, no one has good enough docs.

One problem I've witnessed related to documentation about on-call issues is the over reliance on the SOP concept. They only commit to one level or one pass of analyzing the issue. They do not future drill down, either by linking to other notes or reviewing the issues deliberately. It's like they read about the 5 Whys and decided why not just 1 why.

Re: Being on call sucks

#72

Post by zeckalpha »

> Last year as part of my move to product management, I was removed from the pager rotation for my old team.

Product should be on oncall rotation, too, even if it is as a shadow. This is an important feedback mechanism about the choices they make.

Re: Being on call sucks

#73

Post by killingtime74 »

RajT88 wrote: post #45
dkyc wrote: post #39

Earlier quoted context omitted.

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not…

People are always hiring, don’t sacrifice your health for a job!

Re: Being on call sucks

#74

Post by alistairSH »

cube2222 wrote: post #38
nickjj wrote: post #19

Earlier quoted context omitted.

Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…

You're using "never" everywhere here. That is in my opinion the main red flag here. On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds ha…

almost always there is somebody who's plan is "sitting at home" and nobody minds having the pager in such circumstances

I wonder if this is an American attitude that exists primarily because we (collectively) have allowed our employers to demand this of us?

Personally, I hate being on call. Most weekends, I spend at least 6 hours cycling. Sometimes significantly more. I go camping regularly. I go kayaking regularly. If I don't spend time outdoors, my mental health declines (really, ask my wife, she kicks me out if I sit around the house too long because I turn into a cranky butthead). Most evenings after work, I run/cycle/hike. I have to run errands. Walk the dog (usually 2 miles). This is all part of my normal "not doing anything special" time at home. None of which is easily doable if I'm on call.

Fortunately, I've managed to build a career in a place where on-call rarely exists.

Re: Being on call sucks

#75

Post by eloisant »

ahnberg wrote: post #68

I'm one of the few persons I ever heard of that actually enjoyed being on-call. I believe it goes with my puzzle problem solving mentality to an extent. Being randomly challenged with a problem to look at where you might not know the solution, simply excites me. Combining on-call duty with an approach of weeding out repeating issues, build better systems and ensuring that unnecessary calls don't happen is key of cour…

> And finally having an employer that doesn't expect you to be in at 8am if you've been up all night is also very important, catching up on sleep is necessary to manage your balance and health. But given this freedom, I totally dig it. :)

Check the labour laws in your country, I'm pretty sure expecting people to be in at 8am after working on an incident during the night as part of your on-call is illegal.

In any normal country there are laws to ensure employees get enough rest every day.

Re: Being on call sucks

#76

Post by ricardobayes »

After being a SWE for a decade in Europe, I have never heard of anyone in my network who needed to be on-call. Is this a US thing or only for devops? Why would a software engineer need to be on call ever? That just means the CICD/testing/validation pipeline sucks.

Re: Being on call sucks

#77

Post by sethammons »

My last gig, on call worked well, I thought, for a few reasons: it was our services that we wrote, it was 1 wk out of 6 that you were on call, we heavily prioritized fixing unactionable alerts and automating fixes -- every alert had a runbook entry that described the non-automated fixes, and while on call your or sprint commitments were not counted,

That last point was very nice as it meant you could work on whatever you felt was most important for quality of life improvements all week long while not fielding on call issues. This meant that I looked forward to on call.

Re: Being on call sucks

#78

Post by femiagbabiaka »

jon-wood wrote: post #15

There are many issues with being on-call, particularly in environments where false alerts routinely happen, and where management aren't in the rota so don't directly feel the pain. One of my biggest though is the concept of a weekly rota, with people being on-call for a full week at a time. Sometimes that works fine, and you'll get no alerts all week, but incidents tend to cluster. If something has changed that cause…

Yeah I have a vivid memory of hysterically crying after getting less than 12 hours of sleep over 3 or 4 days during a particularly bad shift. My pager went off 30 minutes before the end of my shift and I lost it. I should’ve passed the pager but I was too “up” to recognize it and my manager at the time was completely clueless about oncall work. Actually at that same company getting paged 8-10 times off-hours during an oncall shift was not unheard of, the burnout was real… I’m surprised there aren’t more lawsuits regarding this because the toll of sleep deprivation on health is well known.

Re: Being on call sucks

#79

Post by jrochkind1 »

jon-wood wrote: post #15

There are many issues with being on-call, particularly in environments where false alerts routinely happen, and where management aren't in the rota so don't directly feel the pain. One of my biggest though is the concept of a weekly rota, with people being on-call for a full week at a time. Sometimes that works fine, and you'll get no alerts all week, but incidents tend to cluster. If something has changed that cause…

How often does someone get a daily rotation though? One week out of four is not uncommon. One out of every four days on-call would really destroy your life though.

Re: Being on call sucks

#80

Post by kazen44 »

morelisp wrote: post #22
stevekemp wrote: post #13

Earlier quoted context omitted.

The Finnish collective agreement, which covers IT workers, spells this out explicitly: * For every hour you're available for on-call you get your regular hourly rate. * For every incident, which involves you working, you receive twice your hourly salary for the duration. So companies based her tend to have that as standard, though I'm sure some companies would pay more to stand out.

This is incredible; at our German company we get 0.1x/1.5x. (Though we also don't promise response times that would interrupt e.g. grocery shopping.)

having resonable response times make a huge difference in being oncall.

in my experience, most employer seem to think responding within 30minutes or so is doable, and i tend to agree with them. Usually makes it possible for you to just do things around the house/ in your life, and not having to rush back home when you run into an oncall issue.

Post reply on HN