Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oncall shift should be Tuesday to Tuesday

arthur-johnston.com

Post reply on HN
Print view View original

131–140 of 230 posts

Re: Oncall shift should be Tuesday to Tuesday

#131

Post by convolvatron »

stackskipton wrote: post #118
convolvatron wrote: post #107

Earlier quoted context omitted.

on call is like hiring civil and structural engineers to build you a bridge over a canyon, and then when they show up to do a site inspection you just push them in. eventually maybe you'll be able to cross.

As SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic. Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed b…

firstly, on call means supporting the entire service. not something in general I built.

secondly, many if not most of the issues that arise are part of some infrastructure automation or third party service or database. expecting me to be fluent in all of those to be useful in the hot seat is a pretty substantial investment and qualifies me to be an SRE on top of my other duties

thirdly, one major reason why my code might fail in production is that it wasn't sufficiently tested, probably because the service as a whole is basically untestable, and even if it were, building test and test infrastructure is likely not at all valued. in many places just filling in that hole would take a year.

onto to the fourth, the story is supposed to be that by operating the service, I'll be incentivized to fix automation and come up with solutions to make it more robust. I actually know how to do this, and every week I'm on call is time that I _dont_ spend doing this. furthermore, getting permission to do so is often like pulling teeth. sounds complicated. sure that would be nice, look at that when you have time in the indefinite future.

so what this often looks like from a development perspective is that I'm being paid to be a developer, I was judged based on my ability to be a developer, but at the end of the day I'm not building the service. I _am_ the service.

Re: Oncall shift should be Tuesday to Tuesday

#132

Post by stackskipton »

convolvatron wrote: post #131
stackskipton wrote: post #118

Earlier quoted context omitted.

As SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic. Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed b…

firstly, on call means supporting the entire service. not something in general I built. secondly, many if not most of the issues that arise are part of some infrastructure automation or third party service or database. expecting me to be fluent in all of those to be useful in the hot seat is a pretty substantial investment and qualifies me to be an SRE on top of my other duties thirdly, one major reason why my code m…

If you are on call for infrastructure, then I could understand not wanting to be on call. If I'm there, I'm on call for infrastructure as SRE.

I get all political reasons that your code may not work. However, refusing to be on call doesn't fix any of those reasons, it's just ignoring work. Flip side as SRE, I ask if Devs are on call. If they are not, I don't take the job because there is zero incentive for them to fix anything vs churn out 5 features, chuck it over the fence and be like "Ops problem now"

Re: Oncall shift should be Tuesday to Tuesday

#133

Post by skuzye »

alienchow wrote: post #74
thuanao wrote: post #10

Is anyone getting compensated for being on-call? If you are paged and work outside of business hours, do you receive additional compensation?

Google paid SREs 67% their hourly rate for tier 1 oncall outside business hours, regardless of whether they were paged. So 12h shifts on weekends were a full day's pay. Convertible to Off-in-Lieu. I had so many off days. Not sure if it's still the case after those layoffs. Anyway, I prefer Mon - Thu, Fri - Sun shifts.

Still the case. It’s a good system IMO. My team is low toil and low page it’s basically free money/time off

Re: Oncall shift should be Tuesday to Tuesday

#134

Post by bleuarff »

Here in France, we have strict laws, and on-calls MUST be paid in some form or another. When we were bought by a US company, mgmt tried to set up on-call shifts for us - we had never needed them for the 10 years prior -, until they learnt of labor laws and went "fuck it, you're on call mon-fri, 10am - 6pm". I'm forty, have a family, and no amount of money could justify that I can't shutoff my phone at night, or prevent from going on a walk on weekends because "uptime". I've never been so glad of french worker protections.

Re: Oncall shift should be Tuesday to Tuesday

#135

Post by hn_go_brrrrr »

rufus_foreman wrote: post #129
deathanatos wrote: post #125

Earlier quoted context omitted.

You expect to not be responsible for what happens to the software you put into production? (… and I'd like to avoid distracting arguments that amount to "my company does on-call badly" — yeah, those problems do exist and we should strive to fix them. But if I'm to not categorize the argument here as the baby with the bathwater, then we need something to replace on-call with. Prod goes down on a Saturday afternoon; ar…

>> You expect to not be responsible for what happens to the software you put into production? I'm responsible for the software I put into production from 9 AM to 5 PM for about 200 days a year. At 3 AM, I am responsible for taking care of myself by getting a good night's sleep. If you need 24 hour coverage, taking into account vacations and weekends, you need 5 or 6 people.

"you need 5-6 people" is moving the goalposts. The root comment said nothing about minimum team size.

Re: Oncall shift should be Tuesday to Tuesday

#136

Post by CoffeeOnWrite »

convolvatron wrote: post #131
stackskipton wrote: post #118

Earlier quoted context omitted.

As SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic. Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed b…

firstly, on call means supporting the entire service. not something in general I built. secondly, many if not most of the issues that arise are part of some infrastructure automation or third party service or database. expecting me to be fluent in all of those to be useful in the hot seat is a pretty substantial investment and qualifies me to be an SRE on top of my other duties thirdly, one major reason why my code m…

> but at the end of the day I'm not building the service. I _am_ the service.

I agree. For me though, it gives me pride to own my services and be fully accountable to the business, especially as part of a team with whom I build comradery, and of course our value to the business justifies our good compensation. It only works because we are empowered to make decisions that keep our on calls sustainable.

Re: Oncall shift should be Tuesday to Tuesday

#137

Post by majormajor »

0xbadcafebee wrote: post #124
sed_zeppelin wrote: post #104

I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated. I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign. There is absolutely no situation in which…

I am 40 yrs old. I get paid a shit-ton of money (just around $200K) to do this stupid tech work job. I work 40 hours a week, I get benefits, flex time, plus I work remote. If I'm getting paged for a legitimate issue that is related to something I built or maintain, then, yes, I am going to respond on-call. Because it's a fucking privilege to get paid this much money to sit on my ass and type into a screen. If I'm get…

An on-call rotation without sufficient influence over the roadmap and planning to be able to fix persistent problems so they don't repeatedly cause the same issues over and over and over is toxic. And it's gonna kill the team's overall productivity so it's not good for management either. Congrats, you're playing SWE salaries for an ops team that would traditionally cost you less otherwise.

In a more healthy situation an on-call rotation is the price of being able to move quickly, get stuff out the door, and have compensation that reflects that the company isn't paying a whole team of extra people to stare at dashboards 24/7 just for the rare situations that things break after-hours.

Gigs with low-overhead + customers that don't expect 24/7 operations are kinda the real sweet-spot dev compensation + role-wise, but ... pretty rare.

Re: Oncall shift should be Tuesday to Tuesday

#138

Post by benhurmarcel »

polack wrote: post #26

We do Thursday to Thursday and then you get Friday off after completed on-call. Being on-call gives you no extra pay by itself, but if you get paged off hours and need to work you get paid 150 to 200% of your normal hourly wage depending on what time of day you need to work. Best on-call I’ve had.

> Being on-call gives you no extra pay by itself

If I’m not paid to be reachable, nobody gets to complain when I don’t pick up the phone though.

Re: Oncall shift should be Tuesday to Tuesday

#139

Post by SoftTalker »

vineyardmike wrote: post #62
bckr wrote: post #28

> But websites need to be up 24/7, cron jobs need to run on the weekend and backend servers need to be up to support both Tech entrepreneurs should give more weight to choosing markets that don’t require this

It used to be somewhat common for websites/services to go down for a few minutes every so often for maintenance/migrations/etc. Tech entrepreneurs should give no weight to this. The market seems to support engineers doing on-call rotations, and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing. Tech entrepreneurs should stop…

It's still common. Nobody really cares if a site is offline for a few minutes. You try again later (or not, but so what). Heck, nobody cares if they are offline for half the day, it gets fixed and at the end of it it's just a post-mortem for the nerds to read and a shrug and life goes on for everyone else. People vastly overestimate the importance of anything that is on the public internet. None of it is life-critical (if it is, it certainly should not depend on an internet connection or a web server being up).

Re: Oncall shift should be Tuesday to Tuesday

#140

Post by badgersnake »

stackskipton wrote: post #118
convolvatron wrote: post #107

Earlier quoted context omitted.

on call is like hiring civil and structural engineers to build you a bridge over a canyon, and then when they show up to do a site inspection you just push them in. eventually maybe you'll be able to cross.

As SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic. Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed b…

Give them the time and budget to build it like a bridge then. Oh wait, your competitors beat you to market by several years.

Making people work 24/7 is not conducive to good anything, thus on call is a terrible way to do things.

Post reply on HN