Being on call sucks • Hacker News Board

Re: Being on call sucks

#261

Post by pm90 » Thu, Jul 21, 2022, 3:59 AM UTC

morelisp wrote: post #199

piva00 wrote: post #25

Earlier quoted context omitted.

> Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second. And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep. It's happened to me, I felt awfully guilty fo…

As someone who is second/third-tier on-call for most of the year: You know what's worse than getting paged when first-tier doesn't handle it? Not getting paged. The only person I ever had to formally reprimand for on-call policy wasn't the one who failed to answer on time, it was the person who, three times, acked and didn't say or do anything else only because they didn't know how to fix it. In all cases I ended up…

I would humbly suggest that if this person didn’t do anything, either they needed to be better informed of expectations or that your culture needs to change so that they don’t feel the need to ack without actually doing anything.

Re: Being on call sucks

#262

Post by pm90 » Thu, Jul 21, 2022, 4:06 AM UTC

The trick to fixing OnCall is to have strong Government regulations around availability of employees to be available off work hours. If that happens, employers will either drop or relax OnCall requirements, or build processes to manage it (e.g. larger teams, dedicated support teams etc).

It’s really that simple. Security professionals tried to get organizations to implement password requirements and MFA for several years, nothing happened. Then SoC2 and other certifications came along, boom, 2FA everywhere.

Re: Being on call sucks

#263

Post by morelisp » Thu, Jul 21, 2022, 5:55 AM UTC

pm90 wrote: post #261

morelisp wrote: post #199

Earlier quoted context omitted.

As someone who is second/third-tier on-call for most of the year: You know what's worse than getting paged when first-tier doesn't handle it? Not getting paged. The only person I ever had to formally reprimand for on-call policy wasn't the one who failed to answer on time, it was the person who, three times, acked and didn't say or do anything else only because they didn't know how to fix it. In all cases I ended up…

I would humbly suggest that if this person didn’t do anything, either they needed to be better informed of expectations or that your culture needs to change so that they don’t feel the need to ack without actually doing anything.

This is also good advice. I just wanted to emphasize that if the higher-tier process was never to be used, it rather wouldn't exist.

Re: Being on call sucks

#264

Post by hyperman1 » Thu, Jul 21, 2022, 7:31 AM UTC

tharkun__ wrote: post #256

hyperman1 wrote: post #251

Earlier quoted context omitted.

Sorry, been there done that. Layout of the file is dictated by an external party. Empty or dummy file is major bad news that blocks other payments. If the file exists, it should have at least 1 valid payment. Payment of 1 cent to a dummy account is illegal. File is produced and consumed by 2 different software packages from 2 different vendors. If the business knowingly creates an invalid file, they commit fraud. To…

I never said 1 cent payment. I didn't know diddly squat about what your domain was or any specifics. You just said that that was a good case of 'bad monitoring is OK'. It's not. Maybe your hands are tied, I'll give you that. It won't make it good though. Also I said 'this space intentionally left blank'. Whatever actual form that would take in your example. Just because multiple entities use the same interface does n…

To be clear: Not a good case of bad monitoring OK. More an unresolvable case given the constraints. I've had to tame plenty of cases of bad interfaces, and that was the main one that evaded any decent resolution.

Re: Being on call sucks

#265

Post by 4ggr0 » Thu, Jul 21, 2022, 9:05 AM UTC

PeterisP wrote: post #162

4ggr0 wrote: post #133

Earlier quoted context omitted.

This sounds like a very software-engineery opinion. Not saying that you're wrong, I understand that on-call is not that important in software development. But as a System Administrator I think that on-call is useful, else we wouldn't notice any outages happening during the night and would then only start working on it in the morning. Plus, we have customers who actually work during the night, be it timezones or speci…

IMHO if you want 24/7 monitoring, then you need 24/7 staffing. If you try to offer 24/7 service and sell SLAs without actually having support people working 24/7 shifts and faking it through on-call, that's the sign of a poorly run company who is effectively lying to their customers (by saying you offer 24/7 service when you really don't) and lying to their employees (by saying that they are working a regular-hours j…

> IMHO if you want 24/7 monitoring, then you need 24/7 staffing

I've never worked at a company which had 24/7 staff, I guess that would require international teams.

> If you try to offer 24/7 service and sell SLAs without actually having support people working 24/7 shifts [...] that's the sign of a poorly run company

The reason I almost can't agree with this is, that almost everyone I know who's working in IT Engineering/Administration works in on-call. This includes people working at the federal post, railways, biggest banks, largest grocery stores, national TV/Radio etc. And that's all in Switzerland, so I doubt that these companies are all poorly run.

It's also not really lying to customers, because customers know about on-call. They know that people generally fix stuff during business hours, with on-call people watching out for the systems during non-business hours.

I wonder if it's a cultural difference or something. I rarely hear people complain about on-call. Not many especially like it, but it's considered to be a thing you just have to do. Also, everyone I know gets paid extra for on-call, and it being Switzerland, on-call has lots of rules.

If you're interested, here's a webpage where the government writes about on-call and all of its rules and definitions, https://www.seco.admin.ch/seco/en/home/Arbeit/Arbeitsbedingu...

Re: Being on call sucks

#266

Post by adastra22 » Thu, Jul 21, 2022, 10:49 AM UTC

4ggr0 wrote: post #265

PeterisP wrote: post #162

Earlier quoted context omitted.

IMHO if you want 24/7 monitoring, then you need 24/7 staffing. If you try to offer 24/7 service and sell SLAs without actually having support people working 24/7 shifts and faking it through on-call, that's the sign of a poorly run company who is effectively lying to their customers (by saying you offer 24/7 service when you really don't) and lying to their employees (by saying that they are working a regular-hours j…

> IMHO if you want 24/7 monitoring, then you need 24/7 staffing I've never worked at a company which had 24/7 staff, I guess that would require international teams. > If you try to offer 24/7 service and sell SLAs without actually having support people working 24/7 shifts [...] that's the sign of a poorly run company The reason I almost can't agree with this is, that almost everyone I know who's working in IT Enginee…

They don’t have a concept of “shifts” in Switzerland?

Re: Being on call sucks

#267

Post by adastra22 » Thu, Jul 21, 2022, 11:00 AM UTC

morelisp wrote: post #182

dml2135 wrote: post #114

I've been at my first software engineering gig for 4 years now, we don't have on call, and I've sworn to myself that I will absolutely never take a position with on-call. I'm a bit worried that it will hamper my career prospects, especially as I've moved to doing more backend work. But I just can't imagine being tied to a work phone on my personal time -- I have a hard enough time enforcing work/life balance as it is…

> I'm a bit worried that it will hamper my career prospects, So honestly, it probably won't, depending on how far you want to go. But lots of the broken teams people are describing here are easily fixed by mandating that the tech lead / architect / whatever participate equally (or more-than-equally) in the on-call process. It's the simplest way to get people with leverage to push back on POs / executives who see it a…

Or work on a product that isn’t a 24/7 hosted service, and enjoy your evenings and weekends.

I’ve never been on call, never will be, and it hasn’t impacted my employment.

Re: Being on call sucks

#268

Post by barking_biscuit » Thu, Jul 21, 2022, 1:12 PM UTC

PeterisP wrote: post #244

niij wrote: post #241

Earlier quoted context omitted.

and that's great in theory, yet incredibly vague. I doubt it's targeting HCI that are predominantly on-call: medical professionals, infrastructure engineers, programmers, etc. I picture it as critical of 996 working hours and similar. In any case I suppose we have different expectations on what's reasonable. If you think a handful of pages per year from my consentual employer is infringing on my fundamental human rig…

I'm looking at this exactly from the perspective of jobs like medical professionals, infrastructure engineers - I have worked in power infrastructure and have relatives in medicine - where locally none of them are on-call (because the law proscribes this clear boundary); they do 12-18-24 hour shifts of work time, and once the shift ends, they can be (and often are) unreachable. It's not vague, it's extremely clear an…

I feel the same.

Re: Being on call sucks

#269

Post by kayodelycaon » Thu, Jul 21, 2022, 1:21 PM UTC

henryfjordan wrote: post #220

kayodelycaon wrote: post #70

On-call is even worse for people with disabilities. I quite literally can't do it unless I stop taking my antipsychotic. Under ADA, I can not be placed on call, regardless of policy, nor can I be discriminated against for that. On-call is not an essential function of being a software developer, with very few exceptions—all of which have nothing to do with "policy" or "fairness". Needless to say, companies (and some c…

Unless you've tested this theory in court it might not be true. It's almost certainly not as cut and dry as you make it seem. Many companies put the same people oncall who write the code, meaning it literally is an essential function of a software developer to provide oncall support. You'd have to argue in court that it's not really essential but it would be situationally dependent. That said, I'd hope most places wo…

That would be one of the exceptions. For example, high-frequency trading firms always need developers on call while they are actively operating. Keeping those systems running correctly is essential to their role in the company. Same with small companies who have no other staff, assuming they even have enough employees (15) to be under ADA. :)

For the more common scenario of on-call rotation, it would be very difficult to make that argument because other people can take up the disabled person's shifts.

Re: Being on call sucks

#270

Post by barking_biscuit » Thu, Jul 21, 2022, 2:19 PM UTC

parasense wrote: post #102

Story time: This one time I was hired as a tier-3 Unix & Linux support engineer at managed-hosting provider who shall not be named. This was back in my prime, I'd already had 10 ~ 15 years industry experience, so I was hired to deal with the really nasty problems that percolated up the chain, and to solve those problems via ongoing process-improvement loop. Over time my job got easier and easier, because lower tier e…

>I was on a two person team, we traded the on-call duty every other week. One day the other person left the company, and that began a comedy of errors...

I knew when I read "I was on a two person team" that you were about to get royally fucked. Been there myself.

I'll never forget so long as I live that "two is one and one is none". I now have a minimum team size of at least 4 developers. Simply not comfortable with anything less.