Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being on call sucks

bobbiechen.com

Post reply on HN
Print view View original

161–170 of 282 posts

Re: Being on call sucks

#161

Post by mcronce »

sethammons wrote: post #77

My last gig, on call worked well, I thought, for a few reasons: it was our services that we wrote, it was 1 wk out of 6 that you were on call, we heavily prioritized fixing unactionable alerts and automating fixes -- every alert had a runbook entry that described the non-automated fixes, and while on call your or sprint commitments were not counted, That last point was very nice as it meant you could work on whatever…

Honestly, that last point seems like something that would make on-call extremely palatable. Essentially acknowledging "on-call sucks; in return, here's the latitude to work on whatever you happen to think is important/interesting"

Re: Being on call sucks

#162

Post by PeterisP »

4ggr0 wrote: post #133
jddddd wrote: post #123

Earlier quoted context omitted.

It won't. On call is a symptom of poorly run company. It's a great signal that you should run far away from any place that requires it. Most software isn't as critical as we think, and the software that is, is expensive enough to have a properly sized staff. On call exists for the same reason game devs are paid shit and open source exists... Software engineers don't value themselves properly and love giving away free…

This sounds like a very software-engineery opinion. Not saying that you're wrong, I understand that on-call is not that important in software development. But as a System Administrator I think that on-call is useful, else we wouldn't notice any outages happening during the night and would then only start working on it in the morning. Plus, we have customers who actually work during the night, be it timezones or speci…

IMHO if you want 24/7 monitoring, then you need 24/7 staffing. If you try to offer 24/7 service and sell SLAs without actually having support people working 24/7 shifts and faking it through on-call, that's the sign of a poorly run company who is effectively lying to their customers (by saying you offer 24/7 service when you really don't) and lying to their employees (by saying that they are working a regular-hours job when really they're doing 24+hour work shifts).

Re: Being on call sucks

#163

Post by jamal-kumar »

I used to do incident response against hacking attempts, DDoS attacks, threats like extortion... Like all of these on a regular basis. Something new and extremely stressful every month or so. Here's a fun fact: when your adversaries are on the other side of the planet they get to wake up early and start their day with the full knowledge that you may be just passing out at like 9pm in your locale. I used to answer my phone with "WHO THE FUCK ARE YOU AND WHY THE FUCK ARE YOU CALLING ME" at hours past that at first. I don't really know how I got used to it (Though living somewhere where stuff is happening 24/7 anyways helped), but the compensation was decent.

Re: Being on call sucks

#164

Post by HelloNurse »

sokoloff wrote: post #125
sam0x17 wrote: post #100

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

What I find interesting is that if you give employees a choice of 100 units of currency per year as salary plus 1 unit of currency for each week on call with an expectation of 1 week on call every four weeks versus 115 units of currency for the year with the same on-call expectations, some employees will feel better about the first arrangement and some will feel better about the second arrangement. Some people want t…

What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive.

There should be no incentive to require or avoid "free" work, only advance planning of who'll be on call (with more available engineers doing more turns).

Re: Being on call sucks

#165

Post by muttled »

Broken_Hippo wrote: post #33
id wrote: post #24

Earlier quoted context omitted.

>It means if you're at your mom's funeral giving a eulogy you leave mid-speech to address PagerDuty. You should not be on-call while you're at your mom's funeral, period.

While I agree totally that you Shouldn't be, we all know that some places will try to force this on you.

I had an awful manager who insisted I needed to "see if I can find someone to cover" for me for a funeral.

Re: Being on call sucks

#166

Post by cube2222 »

throwk8s wrote: post #106
cube2222 wrote: post #6

On-call is just fine as long as: 1. There is a rotation. 2. There are few pages, preferably the median should be 0 per week. 3. Spurious / non-actionable alerts get fixed right away (with very high priority) 4. You're not up more than 1 week per 1-1.5 month. 5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime. As wit…

> 4. You're not up more than 1 week per 1-1.5 month. That seems excessive if you're expected to be able to log into your work system within X minutes. Having to be essentially home, near a computer, 25% of the time (1 week out of 4) is a pretty heavy burden, especially for people who prefer to be out, rather than home.

I just carry around a laptop in my backpack when I'm on call.

Re: Being on call sucks

#167

Post by mceachen »

Aeolun wrote: post #138

> Does anyone have One Weird Trick™ to fix it? My one weird trick is to have a zero tolerance policy for flaky monitors/tests. If it’s not accurate, we either have to drop everything else and fix it, or disable that alarm entirely. Like they say, normalization of deviance is real, and the only way to fight against it is to have every form of deviance be a problem.

This is the correct answer.

For extra credit: if you have a weekly or monthly team meeting, include an agenda item for the people that were on call so they can debrief the team on what alerts fired and what the resolutions were. As a team, you can then decide which alerts need to be deleted or need adjustments, and if there are additions or edits that need to be made to the runbook.

A big thing to avoid in this whole process is "naming and shaming." The Google SRE book calls this a "blameless postmortem culture," and it's helps you avoid perverse incentives for people to hide or obscure latent production issues.

Re: Being on call sucks

#168

Post by 725686 »

bpodgursky wrote: post #116
sam0x17 wrote: post #100

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

Unless you're in a really dysfunctional startup, the employees who voluntarily jump in after-hours to fix OC issues are going to be very quickly compensated with raises and refresher option grants. Far better payout generally than a few bucks of overtime. It becomes really obvious who is holding the company together and who is coasting, relying on the "senior" (ie, anyone who puts in the effort) engineers to keep the…

I don't know for startups, but for regular companies... not. You must be very young my friend.

Re: Being on call sucks

#169

Post by muttled »

5350-uiop-1130 wrote: post #84
Tallianar wrote: post #16

My company(UK) recently tried to force on-call on all engineers. The initial wording was very restrictive, like 5 minute acknowledgement time and 15 minutes at-laptop. 24/7 for 7 days. They tried to have this implemented without any extra remuneration or perks for the on-call engineer. On top of it possibly being very illegal, it seems very immoral to spring something like that on people that did on agree to it when…

why are people so inconsiderate or callous. it's like as long as its not me, i dont care how much your suffer.

I used to work for an MSP. They billed 2-3x the normal rate for on-call to clients. We, however, were simply paid our hourly rate plus overtime. It created a perverse incentive to have as many on-call events as possible as it was very profitable for the company. They billed minimum time to clients, but we were told we could only bill for the exact minutes spent working.

Re: Being on call sucks

#170

Post by bogomipz »

I'm consistently surprised by the number of startups that have distributed teams with people dispersed around the globe and then have an on-call rotation of one person having on-call 24/7 for a week straight. This to me is a management red flag.

One of the first question I used to ask when interviewing for roles that had an on-call component was "what's the on-call expectancy?" I would recommend asking questions about the "on-call" experience to everyone on your interview loop as well. This is often very informative.

Expecting someone to have no life outside of work 24/7 for one week every 5 or 6 weeks is a real quality of life issue. And to do so without offering extra compensation just seems exploitive.

Post reply on HN