Breaking Up with On-Call • Hacker News Board

Re: Breaking Up with On-Call

#101

Post by hylaride » Tue, Mar 18, 2025, 1:55 PM UTC

hbsbsbsndk wrote: post #73

cdnthrownawy39 wrote: post #48

Earlier quoted context omitted.

My on call experience required that I had to be able to respond within 10 ten minutes of the call, with 24/7/365 coverage. But if I couldn't get the issue resolved remotely it meant that I'd have to be in the office lab to recreate and reproduce the problem. It effectively restricted my movements personal movements to stay within commute distance of my office, and that includes all my vacation time as well. That was…

> Obviously people out there that can handle this lifestyle, but I couldn't No I don't think anyone could. What you're describing is an insane policy of abusing employees. Most on-call is done on a rotation, not permanently. All of my previous roles involved on-call and it was 1 week per quarter.

I've worked at places where I was the only person on-call (except during vacations - that's horseshit).

It was fine, but as an SRE/DevOps person I had the authority to override the developer backlogs to get issues fixed and enforce monitoring, architectural, and quality standards. Technical debt rarely built up and we could take one step back before taking two forwards. The product people hate me at first, but the result was usually a platform where problems only ever occurred during deployments (which I've been mostly lucky that I was at places we could do it during business hours) or external factors happened (cloud resource issues, etc).

At my current company, I can count on one hand how many times I've been paged out of business hours in the last 4 years (we recently re-did oncall because it was recognized that it was a risk that I'd become irreplaceable, which is the one point I tried to hammer on them for awhile).

The worst on-call experience I ever had was working at a managed services company. That was the hell of supporting customers that refused to properly allocate resources or invest in their tech stacks. Never again.

Re: Breaking Up with On-Call

#102

Post by happymellon » Tue, Mar 18, 2025, 1:55 PM UTC

hylaride wrote: post #96

ekimekim wrote: post #65

Earlier quoted context omitted.

When I'm in charge of an on-call rotation I always try to make it very clear that this is not the expectation. In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager). The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that…

This is pretty much how it should be done. If the business demands more, they should have a properly manned 24x7 NOC. You also need *ownership*. There is nothing worse than having to support somebody else's work and not being allowed (either via time or other restrictions) to do things "right" so that you're not always paged for fixable problems. Everywhere I worked where the techs had ownership (which varied from OP…

Almost all the reliability issues I encounter is usually due to constraints ordered by people who don't have to deal with on-call.

Things like, running in AWS but you have to use a custom K8S install so they aren't dependent on AWS.

Using self managed Kafka so that you aren't dependent on proprietary tech.

It all sucks because they are always less reliable and generate their own errors and noise for on-calls.

If they had to deal with phone calls every time there's a firewall issue that had absolutely nothing to do with the application, they would soon change their tune.

Re: Breaking Up with On-Call

#103

Post by re-thc » Tue, Mar 18, 2025, 2:00 PM UTC

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

> all I can do is stay at home and be available

If home is fine then usually all you need is Internet and laptop.

> I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip)

Sounds more like setting expectations and explaining the situation than a "cannot" (maybe except movies).

You can explain to your family for example that you're on-call and may need to leave urgently. I mean e.g. police do that. It's not that uncommon.

You can take runs within a 10 minute distance back home. The route is up to you. You can start by acknowledging on the phone as someone else commented, which would grant you maybe another few minutes.

There are lots of options. It's on you to workaround it. On-call isn't perfect, sure.

Re: Breaking Up with On-Call

#104

Post by ZaoLahma » Tue, Mar 18, 2025, 2:24 PM UTC

yodsanklai wrote: post #94

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

10 min doesn't seem tenable over 24/7. Most likely you need to run errands and so on. In my team, our alerts aren't that critical, I just acknowledge on the phone and make sure I can get to the computer with 30 mn. I take it with me if needed. No, for me, the real pain with oncall is that there are a lot of systems in my team. I understand well maybe 30% of it. I'm clueless about 30%. In between for the rest. I can t…

This is exactly why I gave up a position as a full stack / devops engineer in favor of going back to low level drivers - there were too many unknowns, and far too many unknown unknowns often paired with expectations of prompt (and cheap) solutions to complicated issues.

Technically it was interesting and challenging, but in terms of stress just not worth it. You could pay me twice my current salary and I still would not go back to it. Now I try to place myself as far away from paying customers as technically possible.

Re: Breaking Up with On-Call

#105

Post by andrewaylett » Tue, Mar 18, 2025, 2:34 PM UTC

fzeroracer wrote: post #35

rednafi wrote: post #14

A moderate on-call ritual is a necessary evil. I’ve worked at places that tried to get rid of it with all kinds of automation and playbooks, only to revert back to PagerDuty a few months later. That said, my last workplace completely burned me out with a terrible on-call policy and an absurdly short recovery period. Not to mention, upper management tried to gaslight everyone into thinking on-call was just part of nor…

I don't think on-call is a necessary evil; I think it's the result of managers and leaders not caring about having multiple failsafes instead opting to foist that problem onto engineers via unpaid labor. You can have a system with enough redundancy, ability to rollback and deployment scheduling where any sort of on-call incident is highly rare and low impact. But that requires spending time and money on solving those…

We get time off in lieu for hours worked, and paid (at a fixed rate across the company) for hours where we need to be available to be paged. You don't want to only get paid if you're paged, especially if your services are normally quite reliable.

Fail-safes are one thing, but they don't always kick in when we need them. My team is only paged one or two times a month, and when we are it's probably for something we've not seen before because whatever it was that broke last time has been fixed.

Re: Breaking Up with On-Call

#106

Post by dkdbejwi383 » Tue, Mar 18, 2025, 2:42 PM UTC

spiffytech wrote: post #83

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

Any particular reason you can't handle incidents while out and about? I know it varies by situation. When I've been on call I've been able to mostly go about my life. I just had to keep my laptop close, stay in cell signal, and accept I would sometimes have interruptions (typically brief). We fought to keep them infrequent enough that they didn't ruin our lives.

I do long(ish) distance running as a hobby - it's not feasible to take a laptop out on a two hour run.

If I want to go meet a friend for a drink or food, I have to lug around a backpack, keep an eye on it to make sure it's not stolen. If I wanted to have a beer or wine, I can't because I may need to work at any point.

Favourite band is performing? I suppose you could take a backpack and the laptop to the venue, but again there's a chance it's pinched, and they'll make you check it at the cloakroom for the performance.

Re: Breaking Up with On-Call

#107

Post by andrewaylett » Tue, Mar 18, 2025, 2:43 PM UTC

ekimekim wrote: post #65

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

When I'm in charge of an on-call rotation I always try to make it very clear that this is not the expectation. In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager). The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that…

We want to ack within five minutes, and be at a laptop within 30. So long as I'm within mobile signal when the page goes off, it doesn't really matter what I'm doing — an ack is a button press on a push notification. And I can stay within 30 minutes of my laptop and an Internet connection by carrying said laptop and my phone (with "unlimited" data).

If the primary (paid) on-call doesn't catch the notification, the secondary (unpaid) will be paged. And so on, down a couple more steps, to a senior manager. There's no expectation that anyone other than the primary would actually be available to ack the alert.

Re: Breaking Up with On-Call

#108

Post by whstl » Tue, Mar 18, 2025, 2:48 PM UTC

strken wrote: post #89

whstl wrote: post #56

Earlier quoted context omitted.

Yeah, I worked in a couple companies that had the same requirements. It also meant having to wake up on command in case the phone beeped, being unable to drink in my free time, being limited in which activities I could go to. It was hell. I know people are gonna hit back with "you're doing it wrong", but in this case it's the company doing it wrong, but nobody on HN will go there and tell them.

That's absolutely insane. I've argued with a CEO about on-call burdens being too heavy before, and that was with a roster of three people who were only available 7am to 7pm. What kind of slave driver expects literal 24/7/365 availability? Does that not breach labour laws where you work?

The on-call rotation had escalation, and instead of going to the manager it went to someone else in the team. Since there were only two backend engineers in my team, I was either always 1st or 2nd (clown emoji).

Unfortunately this is a huge hole in German labour law.

I quit there pretty fast, after a couple months. It was a tourism company, so Corona treated them very well.

The other company where this happened was a Content Marketing company that was wiped out by ChatGPT. I didn't do on-call there but the other team did.

Re: Breaking Up with On-Call

#109

Post by dkdbejwi383 » Tue, Mar 18, 2025, 2:53 PM UTC

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

If you're on-call, you're working - it doesn't matter if there's an active incident or not. Unless you're a contractor (in which case, you're unlikely to be on-call) the company you work for pays for your time, not delivery of specific work-items. On-call pay should reflect this.

Re: Breaking Up with On-Call

#110

Post by Anon1096 » Tue, Mar 18, 2025, 3:02 PM UTC

ekimekim wrote: post #65

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

When I'm in charge of an on-call rotation I always try to make it very clear that this is not the expectation. In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager). The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that…

We have a primary/backup setup and I would be pretty pissed if my primary just started going out for movies or a date night during their shift tbh. My job as a backup is to be there for unexpected events, ie they did not wake up or had an accident. Not be on call effectively 2 weeks in a row just because the primary doesn't take it seriously.