Being on Call • Hacker News Board

Re: Being on Call

#121

Post by tharkun__ » Sat, Feb 27, 2021, 12:19 AM UTC

unixhero wrote: post #28

lostlogin wrote: post #24

Earlier quoted context omitted.

Yes - the post describes shift work. Oncall is where you go in, then go home, then go in, then go home, but as you touch your back door, you are called back. You say something regrettable to the caller and then apologise. You’re so tired your almost crying and can’t get your laces done up. You drive in, can’t recall the drive and the you are due back for your day shift in 2 hours.

That's pretty funny. In IT it would be more like just getting phonecalls in the middle of the night and operating Service Now tickets, "Mr John Jackson, it is from mainframe services. We need you to file a ticket for team Z to restart service Bob and server Alice." Then go back to bed, and get well paid for it.

That's nowadays.

Contrast that with my dad. NOC calls at 2am and he has to drive I to work for half an hour. Do a 5 minute fix and drive back.

Since they knew him and that he did good work sometimes they would call him even if he wasn't actually on call. He'd ignore those calls but you still wake up... And one time they actually sent a frickin taxi to our house. So half an hour after we ignore the phone, someone rang the door bell. Taxi from company X.

He never quit that job but he signed up next union representative election and actually got elected too :)

Re: Being on Call

#122

Post by znpy » Sat, Feb 27, 2021, 12:41 AM UTC

tharkun__ wrote: post #116

znpy wrote: post #87

Earlier quoted context omitted.

If you come in at work at 10am you might have as well done that trip to the shop/agency before coming to work.

Unfortunately not because most shops in Germany opened at 10am. It's much better nowadays in that AG least grocery stores are open to 10pm.

then come in earlier, i guess?

Re: Being on Call

#123

Post by potatoz2 » Sat, Feb 27, 2021, 1:22 AM UTC

pc86 wrote: post #78

leokennis wrote: post #72

Earlier quoted context omitted.

In my opinion, it’s almost mandatory for a well functioning software development environment to have developers on call (next to ops people). Bugs that get you woken up in the middle of the night are fixed in no time. Documentation that is lacking, or spaghetti code, both preventing you from quickly fixing an issue instead of spending hours on it? Fixed the next day. If the stakes of writing bad code are low, code qu…

There are absolutely some industries where this makes sense (life-saving health care, national defense, things like Amazon where you literally stop making money when things go down). 90% of developers don't work in these industries or on these types of projects. Hell, I work in healthcare, for a private company, and there is zero reason for any of us to be on call. Patient quality doesn't suffer one iota if our softw…

What they're saying is that if someone else is oncall, you tend not to deal with the quality issues of the system (it's a negative externality). If the system you own doesn't require any on-call at all, then great. But for those that do (let's say your a web developer for nytimes.com), the code and system owners should be on-call, not a separate ops team. Ideally being on-call is something you barely notice because nothing happens.

Re: Being on Call

#124

Post by potatoz2 » Sat, Feb 27, 2021, 1:24 AM UTC

sdevonoes wrote: post #84

pc86 wrote: post #79

Earlier quoted context omitted.

If you've got testing, code reviews, QA, and CI/CD, Friday deployments should be totally fine.

The point is "Friday deployments" are not needed (unless we are talking about hotfixes). What's the use case? Can't it wait until Monday? Why are all in such a rush?

Because then you have double the changes to deploy on Monday. And if you push that logic further (no deploys on Thursday, no deploys after 3pm, no deploys +-3 days around holidays, etc.) you end up with very high density deploys, which are riskier.

Re: Being on Call

#125

Post by fuzzfactor » Sat, Feb 27, 2021, 3:10 AM UTC

unixhero wrote: post #65

fuzzfactor wrote: post #63

Earlier quoted context omitted.

I know what you mean, how small do they make violins anyway? In a 24/7/365 cargo operation for a hundred years the port chemists were on call to open their lab after hours and on weekends & holidays each time our ships came in, to operate the office no differently than when you get a vessel during the regular workday. But during regular office hours you definitely need to be there too. These ships are big and they do…

You deserve a bigger audience. Write more! Great story for my friday afternoon work procrastination. How is it going now? Are you still in this business?

Thanks for the kind words, but be careful what you ask for :)

Actually still in the same business, it's a profession that will always be with me even if I do something completely different next.

After the flood it was clear I would have to close eventually, my lab and my life as I knew it were ruined, I just kept working in the ruins.

Closed my company seven years ago after my surviving equipment finally wore out, and went to an employer where I had already served them as a contractor since the 90's. Like any other place they needed help so I went right in and added an extra $million in capabilites the first year with just the gear they already had. Haven't stopped yet.

Of course there's staff there 24/7 now and I'm a senior scientist so it's a straight day job for me but compared to what I am known to do it's been like retirement to have all nights, weekends, and holidays off, so I've kept up 50 to 60 hr weeks the whole time, just to keep up the pace I've always had. Get more than twice as much accomplished this way, not declining performance which many people understandably find occurs to them.

Didn't stop overcoming operational problems and preparing for exponential growth, but with a few disasters hitting this company, that was not able to be deployed.

So with 12 more years progress in only the last 6 years that totals more than 80 years of progressive accomplishments in only the last 40 years.

Now with covid, economy is down and a bit of our equipment is idle too often, I'll be "retiring" and selling these enhanced solutions better than my employer can do. So I'll be an entrepreneur again since I've reached the point where their shareholders and myself will be better off with me on the outside after I've built about all the foundation that can be done for now.

Some of the people I've been working with in the labs have said I must work like Elon Musk or something.

Definitely not I said, he's much younger, I've been doing it much longer, he works like me.

Re: Being on Call

#126

Post by sershe » Sat, Feb 27, 2021, 3:15 AM UTC

devchix wrote: post #98

I carried a pager for a short while, pager, remember those? On-call weeks were hell, even when nothing happened. It's the expectation of getting the call, and the expectation that a system outside your primary domain of expertise will be down. I wasn't paid enough to weather the stress, and I don't think there's a hard number to suit every one. If I ever had input to how an on-call rotation should be handled, I'd mak…

The usual way to handle the "different system" issue is to have on-calls in each team, so you could loop them in for their stuff or even transfer the ticket and go back to sleep; and on-call-focused documentation within teams so you can have some notion of every component that they are not intimately familiar with.

I hate on-call; it doesn't stress me out but I'm generally very protective of my personal time, especially weekends. However, I actually found my greatest on-call highs were from figuring out issues in other people's stuff that was 100% unfamiliar to me, sometime at 4am. In the most memorable case of Rogue Night-Time CPU Spike I didn't even have source code access :)

Re: Being on Call

#127

Post by sershe » Sat, Feb 27, 2021, 3:23 AM UTC

gambiting wrote: post #40

antihero wrote: post #36

Earlier quoted context omitted.

In what hellish sort of company is 9AM "late"?

I sometimes wonder where the hell most HNers work where 9am is not considered late start. At a warehouse I worked at the shift was 6am to 2pm. And it was great that way as you avoided most of the summer heat. The software company where I work now, 8am is standard, that way you're nicely done at 4pm and have the entire evening to yourself. My wife is also a programmer and usually starts at 7:30am because that's their…

Every single software team I've worked in or heard of in the USA? ;) Usually it's tied to the first meeting, I have a mind-destroying train wreck of a "scrum" 3 times a week at 11am so it doesn't make any sense to start before 11. When I used to wake up early consistently in a different team I tried to move the morning scrum to 7am, or maybe 8am, or maybe come on 9:30am?! But people weren't buying it. Meetings before 10am are only scheduled in extreme circumstance.

Re: Being on Call

#128

Post by steveBK123 » Sat, Feb 27, 2021, 3:45 AM UTC

tharkun__ wrote: post #120

steveBK123 wrote: post #112

Earlier quoted context omitted.

If you've actually never been on call, it's hard to understand the way the body responds without experiencing it really. Having every other weekend cut short as you always need to be near a PC from 4pm Sunday... The sheer terror of being woken up from a deep sleep after a long night of work just as you've finally fallen asleep... Resigning yourself to not sleeping as you finally close out the call, full of adrenaline…

It depends on the environment. I have been in both types. I have been in an environment for about 5 years where we Devs were on call for 7 days every month and a half-ish. This was with a business that saw traffic 24/7 with weekends and off hours being higher actually. I was called at 2am once and a couple times on a weekend and I didn't even have to open my computer. I was able to instruct the NOC in what a safe reb…

Yes, 2 weekend & 1 night call in 5 years is absolutely tolerable.

I was at a shop with clever management that fired faster than they backfilled and left the team in such disarray that the replacements started quitting as well. Ended up from a team of 6 down to effectively me & another guy rotating support. I’m talking about getting woken up once or twice a month at 2-3am to join a live incident bridge & debug systems for 2 hours, while being asked to give an update & ETA every 15 minutes, trying not to wake your wife.. and finishing just in time to see the light in the sky. To then head into the office and deal with more fire drills all day and then get asked why feature releases were behind schedule.

Current org has auto-generating incident channels in slack which seems clever until you are on rotation and there are 5 different incident channels live at any given time with messaging popping in throughout the day as some are resolved in minutes, hours or days.

A lot of this is just management failing the marshmallow test.

Re: Being on Call

#129

Post by steveBK123 » Sat, Feb 27, 2021, 3:49 AM UTC

sershe wrote: post #126

devchix wrote: post #98

I carried a pager for a short while, pager, remember those? On-call weeks were hell, even when nothing happened. It's the expectation of getting the call, and the expectation that a system outside your primary domain of expertise will be down. I wasn't paid enough to weather the stress, and I don't think there's a hard number to suit every one. If I ever had input to how an on-call rotation should be handled, I'd mak…

The usual way to handle the "different system" issue is to have on-calls in each team, so you could loop them in for their stuff or even transfer the ticket and go back to sleep; and on-call-focused documentation within teams so you can have some notion of every component that they are not intimately familiar with. I hate on-call; it doesn't stress me out but I'm generally very protective of my personal time, especia…

I have found in my career that being too good at on-call/L3 escalations only makes them come to you more. Teams that have slopey shoulders and phones that seem to always go to voicemail.. don’t get reprimanded, and your team gets to keep playing hero.

Re: Being on Call

#130

Post by dijit » Sat, Feb 27, 2021, 8:11 AM UTC

ornornor wrote: post #109

dijit wrote: post #29

This seems counter-productive. Humans with sleep deprivation are prone to mistakes (as the author admits). I know "we do what our bosses tell us"- but honestly if it was me I would be informing my boss that this is counter-productive and will cost the company (or in this case the country and it's countrymen). Much better to have staggered shifts or even a small night crew.

> if it was me I would be informing my boss that this is counter-productive I suspect you’re fairly new to the workforce. Try that for a few years, see how little effect it has but how much energy it sucks away from you, and then resign to simply doing “what the bosses want” even if it means setting piles of cash on fire on a regular basis.

I’m not sure what fairly new qualifies as. But I’ve been a sysadmin in various companies and countries for 12 years now. (and a person who is on-call for nearly all of that.)

Maybe it’s a European thing, but my bosses have never shied away from the ugly truth that extracting value from humans requires that the humans are able to function properly. Which includes days off, good work life balance and proper rest.