Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oncall shift should be Tuesday to Tuesday

arthur-johnston.com

Post reply on HN
Print view View original

191–200 of 230 posts

Re: Oncall shift should be Tuesday to Tuesday

#191

Post by bsder »

mikeocool wrote: post #187
bsder wrote: post #177

Earlier quoted context omitted.

> If you go to most hospitals at 2AM and need a specialist of some kind (say a specific type of surgeon), there’s going to be someone in that specialty on call whose going to get paged to wake up, come in, and see you. Only if you're dying. Come in late Friday and you're going to be sitting in a bed until Monday even if your gall bladder is about to explode.

Sorry, you’re right. Doctors have it way easier than software engineers.

Sarcasm simply serves to undermine any valid points that you have.

The point was that "on call" is specifically confined as an expectation only to certain types of doctors or under very urgent circumstances.

In addition, doctors have extra special dysfunctions like "too many hours in a shift".

However, many of these are because doctors also have been fighting various efforts to teach more of them which would enable distributing the required extra labor across more people.

Re: Oncall shift should be Tuesday to Tuesday

#192

Post by quicklime »

mikeocool wrote: post #187
bsder wrote: post #177

Earlier quoted context omitted.

> If you go to most hospitals at 2AM and need a specialist of some kind (say a specific type of surgeon), there’s going to be someone in that specialty on call whose going to get paged to wake up, come in, and see you. Only if you're dying. Come in late Friday and you're going to be sitting in a bed until Monday even if your gall bladder is about to explode.

Sorry, you’re right. Doctors have it way easier than software engineers.

I definitely don’t think they have it easier. They work hard and the stakes are much higher.

But what you’re talking about is a person whose job it is to be oncall. It’s the equivalent of an SRE, rather than a SWE. They’re not doing it because they believe in “you build it, you run it” or anything like that.

Re: Oncall shift should be Tuesday to Tuesday

#193

Post by s1artibartfast »

metaltyphoon wrote: post #183
s1artibartfast wrote: post #97

Earlier quoted context omitted.

If that is what you think they are want, you are seriously confused.

Please enlighten me as to how on call is not free work when companies do not pay for this outside of your regular salary .

1) Salary can include tasks like this.

2) You said they want to. They dont. If you offer same pay for a job with and without it, exactly nobody would choose the job with extra on call duties.

The obvious part you are missing is that people do it because they are paid to do it, and they like money.

Re: Oncall shift should be Tuesday to Tuesday

#194

Post by 0xbadcafebee »

QuarterReptile wrote: post #152
0xbadcafebee wrote: post #124

Earlier quoted context omitted.

I am 40 yrs old. I get paid a shit-ton of money (just around $200K) to do this stupid tech work job. I work 40 hours a week, I get benefits, flex time, plus I work remote. If I'm getting paged for a legitimate issue that is related to something I built or maintain, then, yes, I am going to respond on-call. Because it's a fucking privilege to get paid this much money to sit on my ass and type into a screen. If I'm get…

You haven't lived until you've spent a whole weekend at work rushing to fix a production-limiting issue because the boss doesn't know, though you do, about the other division's production-limiting issue which cannot, under any wildly optimistic circumstance, get done in the next two weeks. Oh, and that weekend is the weekend before Christmas.

Oh, I have so, so many on-call stories. The one of "these other people are making our lives miserable" is hard to deal with, but there are paths you can take to get them to work on it. Sometimes it's just not feasible (or is risky) to get them to take more ownership in the short-term. So it's really important to do your own job to establish all the potential failure paths, and set up lines of ownership, make sure your dependencies have their shit together (performance testing, trend analysis, alerts, limits, runbooks, etc) so that when they do inevitably fail you can push back.

I have never been at a job where on-call was done as well as it could be, and most were/are pretty bad in general. But I could always get changes made to on-call, so that when shit started rolling down hill, it didn't hit me.

Re: Oncall shift should be Tuesday to Tuesday

#195

Post by tharkun__ »

joshuamorton wrote: post #155
Retric wrote: post #145

Earlier quoted context omitted.

I don’t put things in production, the company does. And it’s the companies responsible to deal with problems that show up. 24/7 coverage is expensive and mandating someone is on call 24/7 don’t actually provide it.

This is "companies do on-call badly". For the purposes of this exercise presume that our theoretical on-call process is no worse than Google's SRE structure: You are on-call for a 12 hour shift that is more or less aligned with your waking hours, and you are compensated extra for the time you are on-call outside of normal working hours, whether or not you are called in. You are on-call at most one week per month, on…

    You are on-call for a 12 hour shift that is more or less aligned with your waking hours
I suppose if you're Google they can theoretically make it so it's more aligned with your waking hours? Do they do it? Most companies don't or can't. I.e. it's _less_ aligned.

    you are compensated extra for the time you are on-call outside of normal working hours, whether or not you are called in
How much? Way too many on-call processes in which this is nothing but a few dollars to be able to say "see, we do pay for this, even when you're not called!". As in, way not enough for the number being on-call does to how you go about your day. Always on edge, always awaiting that call / alert that requires you to drop whatever you are currently doing. Preventing you from actually doing/starting certain things.

You haven't even mentioned the expected reaction and resolution time and that alone can make a huge difference.

    You are on-call at most one week per month, on average, and usually less.
Great, only one week out of four /s That's crazy if you ask me. Going back to preventing you from going about your day in a normal way. There's no "doing on-call well" in how you describe it.

Re: Oncall shift should be Tuesday to Tuesday

#196

Post by alienchow »

skuzye wrote: post #133
alienchow wrote: post #74

Earlier quoted context omitted.

Google paid SREs 67% their hourly rate for tier 1 oncall outside business hours, regardless of whether they were paged. So 12h shifts on weekends were a full day's pay. Convertible to Off-in-Lieu. I had so many off days. Not sure if it's still the case after those layoffs. Anyway, I prefer Mon - Thu, Fri - Sun shifts.

Still the case. It’s a good system IMO. My team is low toil and low page it’s basically free money/time off

I agree, it incentivises SREs to reduce pager toil.

Re: Oncall shift should be Tuesday to Tuesday

#197

Post by coffeefirst »

SoftTalker wrote: post #139
vineyardmike wrote: post #62

Earlier quoted context omitted.

It used to be somewhat common for websites/services to go down for a few minutes every so often for maintenance/migrations/etc. Tech entrepreneurs should give no weight to this. The market seems to support engineers doing on-call rotations, and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing. Tech entrepreneurs should stop…

It's still common. Nobody really cares if a site is offline for a few minutes. You try again later (or not, but so what). Heck, nobody cares if they are offline for half the day, it gets fixed and at the end of it it's just a post-mortem for the nerds to read and a shrug and life goes on for everyone else. People vastly overestimate the importance of anything that is on the public internet. None of it is life-critica…

Correct. Things are breaking all the time. If you’re not a hospital or air traffic control, nobody is going to die if your website goes down.

There’s a time and a place for heroics, but we go to it for shit that doesn’t really matter, or worse, allow the culture of heroics to cover up the real problems that are much harder to fix.

Re: Oncall shift should be Tuesday to Tuesday

#198

Post by doubleg72 »

bsder wrote: post #191
mikeocool wrote: post #187

Earlier quoted context omitted.

Sorry, you’re right. Doctors have it way easier than software engineers.

Sarcasm simply serves to undermine any valid points that you have. The point was that "on call" is specifically confined as an expectation only to certain types of doctors or under very urgent circumstances. In addition, doctors have extra special dysfunctions like "too many hours in a shift". However, many of these are because doctors also have been fighting various efforts to teach more of them which would enable d…

Funny, my wife is primary care yet does on call via answering service. You clearly don’t know what you’re talking about.

Re: Oncall shift should be Tuesday to Tuesday

#199

Post by chipsa »

bsder wrote: post #177
mikeocool wrote: post #163

Earlier quoted context omitted.

If you go to most hospitals at 2AM and need a specialist of some kind (say a specific type of surgeon), there’s going to be someone in that specialty on call whose going to get paged to wake up, come in, and see you. Even in family practice, it’s not uncommon to be able to get a call back from the on call doctor at the practice on weekends or off hours — if you’ve got a situation that maybe doesn’t warrant the ER, bu…

> If you go to most hospitals at 2AM and need a specialist of some kind (say a specific type of surgeon), there’s going to be someone in that specialty on call whose going to get paged to wake up, come in, and see you. Only if you're dying. Come in late Friday and you're going to be sitting in a bed until Monday even if your gall bladder is about to explode.

I went in to a hospital at just after midnight, and had my gall bladder out by noon. No, the surgeon wasn’t called in early, but the radiologist who diagnosed the gall bladder was.

Re: Oncall shift should be Tuesday to Tuesday

#200

Post by SpicyLemonZest »

SuperNinKenDo wrote: post #173
nosefurhairdo wrote: post #159

Earlier quoted context omitted.

Is the argument here that every software team should have engineers whose normal working hours have 24/7/365 coverage?

If you expect your team to provide 24/7/365 assurance, then it's hard to see how that isn't a perfectly reasonable idea. The only counter to it is that keeping people on call shifts financial cost off the business in the form of psychological cost to its employees. Not very convincing.

Would you take the night shift? Everyone I've seen promote this idea seems to expect that they'll be the lucky ones who get to keep a normal schedule. If you have a service that needs 24/7 uptime, and you transition from an oncall model to a shift model, at least 2 out of every 3 engineers on the team are going to have to change shifts or quit. If the entire industry shifts, high-availability software would simply join the ranks of fields like nursing or manufacturing where many people have no realistic option to work normal hours.
Post reply on HN