Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oncall shift should be Tuesday to Tuesday

arthur-johnston.com

Post reply on HN
Print view View original

211–220 of 230 posts

Re: Oncall shift should be Tuesday to Tuesday

#211

Post by Spooky23 »

deathanatos wrote: post #125
sed_zeppelin wrote: post #104

I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated. I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign. There is absolutely no situation in which…

You expect to not be responsible for what happens to the software you put into production? (… and I'd like to avoid distracting arguments that amount to "my company does on-call badly" — yeah, those problems do exist and we should strive to fix them. But if I'm to not categorize the argument here as the baby with the bathwater, then we need something to replace on-call with. Prod goes down on a Saturday afternoon; ar…

“Devops” traded less bureaucracy for more accountability.

Have a generalist ops team that is staffed 24x7, or has paid on call as part of the job. They get run books to respond to whatever goes on.

I’ve set this up twice. The first time, we had a team in the Philippines that would cover overnights.

They could start and rollback deployments and do most stuff via the runbook they were provided. Most callouts (5% of escalations) to product teams were due to bad or missing documentation.

The US based team did similar work, just during the day. Both could escalate quality issues for the product team to fix.

The other model was all US, on-call based. We used junior and low-skill folks, who had rotating on-call. They were paid 20% of hourly rate for standby pay and had a minimum pay threshold when they got called. All of that hit the cost center of the offending product or service, so there was both a financial incentive to not get calls, and a human incentive as the engineers didn’t want to get called for escalations. Again, documentation is key.

Re: Oncall shift should be Tuesday to Tuesday

#212

Post by [deleted] »

nosefurhairdo wrote: post #159
mikedelfino wrote: post #150

Earlier quoted context omitted.

If the company has enough people in the team, someone just works the night shifts or on scheduled weekends. No one needs to be on-call because there would be someone taking care of it already.

Is the argument here that every software team should have engineers whose normal working hours have 24/7/365 coverage?

[deleted]

Re: Oncall shift should be Tuesday to Tuesday

#213

Post by s1artibartfast »

metaltyphoon wrote: post #207
s1artibartfast wrote: post #193

Earlier quoted context omitted.

1) Salary can include tasks like this. 2) You said they want to. They dont. If you offer same pay for a job with and without it, exactly nobody would choose the job with extra on call duties. The obvious part you are missing is that people do it because they are paid to do it, and they like money.

> Salary can include tasks like this So lets say that it just magically happens that when YOU are on call, stuff breaks all the time but when its your coworkers it doesn’t. You are all paid the same, does it seem fair to you now? Unless it’s written in paper where a salaried worker will be getting X extra per hour you are just working for free. The definition of a salaried worker in the US is having 40hrs of total wo…

>The definition of a salaried worker in the US is having 40hrs of total work time averaged throughout a year.

I think we got to the heart of things. This is absolutely not true! Not legally, and not in practice. There are Overtime exempt salaried positions and non-exempt positions [1]. An exempt salary position position pays more than $685/week and means you do the "the role" however your employer defines it. That can be 40 hours, 80 hours, or whatever they choose. It can require you live on-site for the whole year.

https://www.dol.gov/agencies/whd/fact-sheets/17a-overtime

Re: Oncall shift should be Tuesday to Tuesday

#214

Post by throwaway2037 »

renewiltord wrote: post #170
sed_zeppelin wrote: post #104

I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated. I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign. There is absolutely no situation in which…

I think the useful thing here is to mention the trade-off. Practically all my SWE friends making $1m+ are instant responders whether denoted so or not.

I know HN doesn't like jokey replies, but you wrote: "all my SWE friends making $1m+", where "friends" is plural. You have multiple friends who earn more than 1M USD total comp per year? Man, HN is getting crazier by the day.

Re: Oncall shift should be Tuesday to Tuesday

#215

Post by thebigspacefuck »

bigiain wrote: post #31
thebigspacefuck wrote: post #30

On a past team I set up on-call to be: - Mon/Tue - Wed/Thu - Fri - Sat/Sun Original reason for this schedule was that on-call was paid by days per quarter in a tiered system so this guaranteed that all members got the 5% on-call for 10 days/quarter rather than one person hitting 9 days and dropping to 3%, but I stand by this as a better on-call rotation. The number of people does need to be not wholly divisible so th…

Any company that makes it an employee's responsibility to find "someone to cover" their on call time while they're on vacation is a company worth quitting. I'm pretty sure that'd be illegal here in .au On call coverage while an employee is on vacation is a management problem, not an employee problem.

Most companies are like this in my experience. You have some default rotation going out to forever and as part of planning vacation you check whether the time you are requesting is when you're scheduled for on-call and if so ask if someone else is available to swap on-call. If you can't find someone to cover, you raise to your manager and either they'll cover it or find someone else to cover.

Re: Oncall shift should be Tuesday to Tuesday

#216

Post by pb7 »

bleuarff wrote: post #134

Here in France, we have strict laws, and on-calls MUST be paid in some form or another. When we were bought by a US company, mgmt tried to set up on-call shifts for us - we had never needed them for the 10 years prior -, until they learnt of labor laws and went "fuck it, you're on call mon-fri, 10am - 6pm". I'm forty, have a family, and no amount of money could justify that I can't shutoff my phone at night, or preve…

[flagged]

Re: Oncall shift should be Tuesday to Tuesday

#217

Post by ksmith14 »

tharkun__ wrote: post #195
joshuamorton wrote: post #155

Earlier quoted context omitted.

This is "companies do on-call badly". For the purposes of this exercise presume that our theoretical on-call process is no worse than Google's SRE structure: You are on-call for a 12 hour shift that is more or less aligned with your waking hours, and you are compensated extra for the time you are on-call outside of normal working hours, whether or not you are called in. You are on-call at most one week per month, on…

You are on-call for a 12 hour shift that is more or less aligned with your waking hours I suppose if you're Google they can theoretically make it so it's more aligned with your waking hours? Do they do it? Most companies don't or can't. I.e. it's _less_ aligned. you are compensated extra for the time you are on-call outside of normal working hours, whether or not you are called in How much? Way too many on-call proce…

Google staffs SRE teams as either 8 in one location/TZ or two geographically distributed teams of 6 -- often some pairwise combination of U.S., Europe, and Australia to accommodate reasonable on-call shifts.

The on-call compensation varies depending on what tier of service they're offering. Tier 1 (5 minute response time) is 2/3 of your effectively hourly pay for on-call time outside of local business hours and 1/3 for tier 2 (30 min response time). Or time off in lieu.

Re: Oncall shift should be Tuesday to Tuesday

#218

Post by srhtftw »

sed_zeppelin wrote: post #104

I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated. I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign. There is absolutely no situation in which…

At last, a voice of reason amid the vulgar crowd.

I've held several management positions where I've carried a pager. At one employer I helped keep trading databases operational in 7 time-zones on 3 continents. At another I helped fix a backup issue on Christmas eve. Helping those customers was a core part of my responsibility. I fully understood that and took great pride in it.

But as a developer I too will never accept another on-call rotation.

Companies which assign on-call duties to developers make the mistake that development, management and operations are different kinds of work which require different environments and skill-sets. Other engineering tasks include testing, documentation, training, and maintenance. At small startups the founders and early employees may do some or all of these but that becomes impractical at larger established businesses.

Engineers should learn and do all these things in the course of their career but not all at the same time unless quality isn't a concern.

My experience at a unicorn a few years ago convinced me companies which assign developers on-call rotation either don't understand or don't care about the quality or sustainability of their business. In that company senior management was replaced by folks from Google and Facebook shortly after I joined. I was moved into a team where I had no role in the design, develop or deployment of its services. I had no say in the hiring or firing of the so-called engineers who rushed failing services into place past a wholly ineffective QA department.

I should have seen the writing on the wall when I began to be pressured by managers and recruiters to rubber-stamp candidates who couldn't pass our coding tests but had spent lots of time on-call. The company's priorities slowly became clearer to me as they grew evermore desperate to live up to their promises. Ultimately I suffered an ischemic attack from the stress of this environment and left the company to focus on my health.

Oh and the company? It let go of most of its engineers a year later and was eventually acquired by competitor for a few hundred million after having raised over a billion dollars.

Re: Oncall shift should be Tuesday to Tuesday

#219

Post by erik_seaberg »

coffeefirst wrote: post #197
SoftTalker wrote: post #139

Earlier quoted context omitted.

It's still common. Nobody really cares if a site is offline for a few minutes. You try again later (or not, but so what). Heck, nobody cares if they are offline for half the day, it gets fixed and at the end of it it's just a post-mortem for the nerds to read and a shrug and life goes on for everyone else. People vastly overestimate the importance of anything that is on the public internet. None of it is life-critica…

Correct. Things are breaking all the time. If you’re not a hospital or air traffic control, nobody is going to die if your website goes down. There’s a time and a place for heroics, but we go to it for shit that doesn’t really matter, or worse, allow the culture of heroics to cover up the real problems that are much harder to fix.

Imagine Walmart telling all their customers "Come back tomorrow, maybe, we can't manage to keep any of our stores open. At least nobody is going to die."

Re: Oncall shift should be Tuesday to Tuesday

#220

Post by sokoloff »

dangus wrote: post #112
hawaiianbrah wrote: post #88

Earlier quoted context omitted.

> If you took time off during an on-call shift you would be trading it with a team member, so you would never lose that extra wage. I think this is true in _most cases_, but is not a given. I myself have encountered scenarios where it isn’t true: switching with someone much later in the rotation, only to then end up having to switch again for instance. You could envision a nefarious teammate weaseling out of their fa…

Of course it wouldn’t be hard to figure out a rough average on-call amount to pay during PTO

At what point along this continuum does it just become "base salary" rather than "pay specifically for being on-call"?
Post reply on HN