Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking Up with On-Call

reflector.dev

Post reply on HN
Print view View original

61–70 of 202 posts

Re: Breaking Up with On-Call

#61

Post by procaryote »

mrkeen wrote: post #46
pokoleo wrote: post #3

From my experience working on SaaS, and improving ops at large organizations, I've seen that "on-call culture" often exists inversely proportional to incentive alignment. When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes. When incident response becomes an organizational checkbox divorced from financial outcomes and pla…

Engineers don't pick their work, management does. A manager no longer needs to choose between system reliability and churning out new features with on-call: The manager can get all the credit for pushing out new features during the day, and sleep well at night knowing that the engineers aren't.

At a lot of companies engineers are involved in picking the work. It's silly to hire competent problem solvers and treat them as unskilled workers needing micro-management.

Besides, if you set the on-call system up so people get free time the following day to compensate for waking up at night, the manager can't pretend there's no cost.

Bad management will fail on both of these of course, but there's no saving that beyond finding a better company.

Re: Breaking Up with On-Call

#62

Post by dakiol »

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip). Basically, all I can do is stay at home and be available. It sucks, and the money is not worth it.

Re: Breaking Up with On-Call

#63

Post by procaryote »

AdieuToLogic wrote: post #26

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system.…

If the code is important enough that it needs to be fixed if it breaks at 03:00, someone needs to be on-call

If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking.

I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call.

Pain from a problem needs to live where the problem can be fixed.

Re: Breaking Up with On-Call

#64

Post by rwmj »

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that.

https://en.wikipedia.org/wiki/Time-and-a-half

Re: Breaking Up with On-Call

#65

Post by ekimekim »

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

When I'm in charge of an on-call rotation I always try to make it very clear that this is not the expectation.

In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager).

The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that's what the secondary role is for - to catch you. This means it's perfectly acceptable to go for a run, go to a movie, etc.

You relax the responsibility on the individual and let a sensible amount of redundancy solve the problem instead. Everyone is less stressed, and sure you get the occasional 5min delay in response but I'm willing to bet that the overall MTTR is lower since people are well rested and happier to be on call to begin with.

Re: Breaking Up with On-Call

#66

Post by chgs »

rwmj wrote: post #64
dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half

Last time I was on call it was a one hour payment per day, a 4 hour payment for answering the phone.

That lasted about 3 months, just not acceptable.

Re: Breaking Up with On-Call

#67

Post by liveoneggs »

rwmj wrote: post #64
dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half

For nurses on-call is a tiny amount - $3/hr and then you get 1.5x if you actually get called in

Re: Breaking Up with On-Call

#68

Post by chgs »

rohgpt wrote: post #54

I used to be on-call too. I still have muscle memory of the buzz that’d make me leap off my couch like my house was on fire. Except it wasn’t. Just prod, doing prod things. We all know it. You trade your peace of mind, sleep, and sometimes personal freedom (hello, “don’t go beyond cell signal” life). And I agree with many of you—it shouldn’t be this way. We’ve seen teams turn their on-call rotations into something wa…

Why don’t you have shift work?

Re: Breaking Up with On-Call

#69

Post by zild3d »

rwmj wrote: post #64
dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half

So for say a weekly on-call rotation you would be paid all 24 hours x 7 days at double rate? (- ~40)

Also most tech companies don't have unions...

Re: Breaking Up with On-Call

#70

Post by xyst »

Question to tech folks in general: when you are "on-call" are you provided with differentials in pay regardless of whether an incident actually occurs?

I have seen it vary across companies:

- Some companies don’t pay any differential

- Some companies only pay out _per incident_

- One company I worked with paid out per shift of on-call rotation

- One company treated it as overtime pay

Post reply on HN