Live data from Hacker News

Breaking Up with On-Call

reflector.dev

121–130 of 202 posts

Re: Breaking Up with On-Call

#121

Earlier quoted context omitted.

10 min doesn't seem tenable over 24/7. Most likely you need to run errands and so on. In my team, our alerts aren't that critical, I just acknowledge on the phone and make sure I can get to the computer with 30 mn. I take it with me if needed. No, for me, the real pain with oncall is that there are a lot of systems in my team. I understand well maybe 30% of it. I'm clueless about 30%. In between for the rest. I can t…

This is exactly why I gave up a position as a full stack / devops engineer in favor of going back to low level drivers - there were too many unknowns, and far too many unknown unknowns often paired with expectations of prompt (and cheap) solutions to complicated issues. Technically it was interesting and challenging, but in terms of stress just not worth it. You could pay me twice my current salary and I still would…

How'd you manage the transition (back?) to low-level? I would love to do chip work (or really anything systems-y) but all my experience is fullstack/webdev. Every time I apply I get bounced for insuffficient domain experience.

Re: Breaking Up with On-Call

#122
I was at the tender age of 21, and a mere contractor at my first job. I recall being paid hourly, but never filling a time card.

I had just been promoted to a systems admin role, after demonstrating prowess with coding and system configuration, and also the actual qualified admin had quit. One of my projects involved a rewrite of server code for efficiency as our /etc/passwd balooned to thousands of users, and too many logging in simultaneously.

So on Thanksgiving Day, we just got in the door with Mom after shopping, and my supervisor rings the home phone; his first words are "There's a hacker on [production shell server]". And I told him a few commands to try, and we did not seem to suffer downtime or need our backups.

That may have been the most consequential job I had for a decade. There were thousands of customers depending on us giving "four nines" of available service, including the corporate customers, but we had no formal on-call agreement, not even a way for me to access systems from outside the office, and I was still a novice after a few college courses and a lot of unstructured hacking.

My [salaried] father, on the other hand, often brought a pager [which never seemed to beep] home, and one day we were all sitting around the living room when an explosion rattled our windows. We were still a bit rattled ourselves when the phone rang, and it was his cue to go down to the site and respond as a health & safety officer.

So my mother said that our household policy would henceforth be: "if something explodes out there, nobody answers the phone!"

Re: Breaking Up with On-Call

#123
post #10

Was author's big tech experience with Amazon? Because Amazon oncall is ... well it's something. But I'm not sure it's really indicative of the rest of big tech oncall.

The author's Linkedin indeed lists working at AWS for 3 years.

Ok that explains a lot about this article

Re: Breaking Up with On-Call

#124
post #92

Earlier quoted context omitted.

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT 's job, whereas I'm SE . No thanks

My wife is part of a union and there’s none of that bullshit. However when her employer wanted to reduce costs across the board the union negotiated a shorter working week for everyone instead of a pay rise next year. They voted on it and accepted it with an overwhelming majority.

The union offering "shrinkflation" as the way for the business to cut costs is an interesting framing. Your wife's union associates must hang out with grocery store executives.

Re: Breaking Up with On-Call

#125

Earlier quoted context omitted.

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT 's job, whereas I'm SE . No thanks

While you will find some extreme examples that could go that far, unions don't generally do that. Organisations that fight unions however do like to bring up that example, so... you've been had with anti union propaganda.

So my coworker who was a UAW member who told me stories about sleeping on the roof, and being reprimanded for moving a desk to retrieve a pen...was trying to dupe me?

Re: Breaking Up with On-Call

#126
post #100

Earlier quoted context omitted.

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT 's job, whereas I'm SE . No thanks

That isn't how unions work, rather the usual US fearmongering of having them.

[deleted]

Re: Breaking Up with On-Call

#127
post #100

Earlier quoted context omitted.

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT 's job, whereas I'm SE . No thanks

That isn't how unions work, rather the usual US fearmongering of having them.

It just dawned on me how this argument runs perfectly parallel to religion if you point out intolerance, misogyny, or violence. It's always "those other people" that do the bad thing, and everyone only reaffirms their own system of worship. You could almost do a 1-1 find/replace of keywords and have the same argument.

Re: Breaking Up with On-Call

#128
post #3

From my experience working on SaaS, and improving ops at large organizations, I've seen that "on-call culture" often exists inversely proportional to incentive alignment. When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes. When incident response becomes an organizational checkbox divorced from financial outcomes and pla…

IMO it's when the incident response and readiness practice imposes a direct backpressure on feature delivery that you get the issues actually fixed and a resilient system. if it's just the engineer while product and management see no real cost then people burn out and leave. > The most successful teams I've seen treat on-call like a leading indicator - every incident represents unpriced technical debt that should be…

The people who do the real work don’t get raises and promotions because the annual review system punishes them for doing the right thing.

Re: Breaking Up with On-Call

#129
post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

> This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip).

I live in rural Texas. The same things apply here, and more: I'm lucky to have good internet (which enables working remotely) but half my home doesn't get cell coverage so being responsive to a text message or phone call means not even going around my own home (for example, no cell signal in the kitchen means I can't cook while on-call); and with large tracts of land, I can't go out to do land maintenance (good luck hearing a phone ring or feeling it vibrate from a call when you're operating heavy machinery, assuming you even have cell signal there); all services are 15 minutes or more away: groceries, doctor, contractors, government, etc etc.

It's important to stress how much being on-call ruins my capability to use the time effectively for my own purposes (Texas Guidebook for Employers [0]; 29 CFR 785.16 [2] and 785.17 [3]). I tried telling this to a previous employer when they started wanting me to be on-call (3+ years after start of employment), and they indicated that those laws are only used for hourly employees but being salary + exempt means I do not qualify for additional pay and falls under "and other duties as assigned" in the employment contract. So the employer effectively started getting 60 hours of work for 40 hours of pay. Oof.

I also absolutely refuse to mix my personal devices with work; just at a minimum, I refuse to make my personal device available to legal discovery related to any legal issues with the employer. So if the employer wanted me to have cell phone availability, then I demanded that the employer provide that cell phone. That was a fun conversation that ended with some relaxed requirements (eg, I don't have to have cell phone availability if I'm responsive at my work desk already) which further reinforced the fact that I couldn't use the time for my own purposes.

Thankfully multiple years in this industry at (what was) fair compensation allows me to be picky for new employment contracts. And lesson learned: I'll be a lot more careful about contract language from now on, and specifically look for (or negotiate) carve-outs around being on-call and work/personal device separation. I recognize that having 10+ years of experience makes me able to handle that, but newcomers to the industry won't yet have that buffer and it sucks for them to not have that safety net for negotiation leverage.

A lot of this disagreement comes from businesses demanding rapid response while insisting on not taking on new hardware/payment obligations. To contrast: take the fireman who's waiting for an alarm (29 CFR 785.15 [1]): they are often often idle and can often go out for groceries but they're easily reachable. Ever seen a firetruck in front of a grocery store and the firemen are just inside shopping for groceries? Then see them come running out and turn on the lights & siren and drive off? I have. It's an interesting event, and it sucks for the grocery store that has to put those groceries (for ~15 people) back on the shelves and refrigerators. Nonetheless, those firemen are paid to do so and have special equipment (eg radios or cell phones) to be able to receive those messages, and the firement generally don't pay for that equipment themselves (the community does either through taxes or donations). I see analogies about on-call software engineers being called to put out (virtual) fires as very apt in this case.

[0]: https://efte.twc.texas.gov/c_waiting_or_on_call_time.html

[1]: https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...

[2]: https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...

[3]: https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...

Re: Breaking Up with On-Call

#130
Here's my answer to on-call requests: no. And if you ask me to clarify my statement, I will answer: no. Because no is good enough, and it's what I negotiated when I took the job.

I don't work for employers who can't manage their staff properly. If you're on-call at a company with any sort of scale, you aren't being managed properly. One of the things I tell hiring managers is that I don't mind working later if we're reaching the end of a project and a late afternoon or two might get the product pushed out of the door sooner, or if I've gotten into the zone and am making reat progress. But these are my choices to make. There will never ever be a time when I will subject myself to being tied to a phone for the purposes of waking up in the middle of the night and messing with servers or software. My time is MY time. I only ever have reciprocal relationships with employers and refuse to upset that balance.

Post reply on HN