Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking Up with On-Call

reflector.dev

Post reply on HN
Print view View original

91–100 of 202 posts

Re: Breaking Up with On-Call

#91

Post by jjice »

AdieuToLogic wrote: post #26

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system.…

The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center".

It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years.

While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big different than the one you've experienced (which may be the more common one, I'm not sure).

So this may end up being a cultural difference between on-call situations.

Re: Breaking Up with On-Call

#92

Post by maccard »

gosub100 wrote: post #90
rwmj wrote: post #64

Earlier quoted context omitted.

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT 's job, whereas I'm SE . No thanks

My wife is part of a union and there’s none of that bullshit. However when her employer wanted to reduce costs across the board the union negotiated a shorter working week for everyone instead of a pay rise next year. They voted on it and accepted it with an overwhelming majority.

Re: Breaking Up with On-Call

#93

Post by dontlaugh »

rwmj wrote: post #75
zild3d wrote: post #69

Earlier quoted context omitted.

So for say a weekly on-call rotation you would be paid all 24 hours x 7 days at double rate? (- ~40) Also most tech companies don't have unions...

I don't believe it's physically possible for anyone to be available at 10 minutes notice for 168 hours straight. And if it's possible, it would be deeply unhealthy. But if they did achieve this, then yes they should be paid some very large amount of money. But that doesn't just happen -- pay isn't fair, it's about power. So a union can help to negotiate this, or ideally, better working conditions. > Also most tech co…

Also in the UK there’s https://utaw.tech/ and https://www.gameworkers.co.uk/

If there isn’t a union recognised in your workplace, you can build one yourself.

Re: Breaking Up with On-Call

#94

Post by yodsanklai »

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

10 min doesn't seem tenable over 24/7. Most likely you need to run errands and so on. In my team, our alerts aren't that critical, I just acknowledge on the phone and make sure I can get to the computer with 30 mn. I take it with me if needed.

No, for me, the real pain with oncall is that there are a lot of systems in my team. I understand well maybe 30% of it. I'm clueless about 30%. In between for the rest. I can try to fix issues myself (take long time, issues can add up), triage (but means bothering someone else). There are also things than nobody understand, and if they break, it can mean extra days of work for you with extra-stress because you don't know how hard it can be to fix.

Then, some people in the team ship code without adequate testing, because of pressure to ship. Which often adds work to the oncall. So there's all this extra tensions with colleagues which can be hard to deal with for an introvert.

Overall, it "kind of" works for us, but I agree with the conclusion, it sucks. It's really the worst part of my job. I went into software engineering because I like coding. Not because I liked monitor unreliable systems. And I think unreliability is encouraged by management to some extent. That keeps people at work.

Re: Breaking Up with On-Call

#95

Post by meeshmuesh »

northern-lights wrote: post #43
pokoleo wrote: post #3

From my experience working on SaaS, and improving ops at large organizations, I've seen that "on-call culture" often exists inversely proportional to incentive alignment. When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes. When incident response becomes an organizational checkbox divorced from financial outcomes and pla…

This assumes that the engineers who wrote the code that caused the 3 AM pages will still be around to suffer the consequences of the 3 AM pages. This is a lot of times, not true, especially in an environment which fostered moving around internally every now and then. Happens in at least one of the FAANGs.

Sounds like Amazon

Re: Breaking Up with On-Call

#96

Post by hylaride »

ekimekim wrote: post #65
dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

When I'm in charge of an on-call rotation I always try to make it very clear that this is not the expectation. In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager). The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that…

This is pretty much how it should be done. If the business demands more, they should have a properly manned 24x7 NOC.

You also need *ownership*. There is nothing worse than having to support somebody else's work and not being allowed (either via time or other restrictions) to do things "right" so that you're not always paged for fixable problems. Everywhere I worked where the techs had ownership (which varied from OPS people being allowed to override the backlog to fix issues or developers being given enough free reign to fix technical debt) has usually meant that oncall is barely an issue. My current gig I often forget I'm even on call at all and the main issues that do crop up are usually external.

Re: Breaking Up with On-Call

#98

Post by cdnthrownawy39 »

spiffytech wrote: post #83
dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

Any particular reason you can't handle incidents while out and about? I know it varies by situation. When I've been on call I've been able to mostly go about my life. I just had to keep my laptop close, stay in cell signal, and accept I would sometimes have interruptions (typically brief). We fought to keep them infrequent enough that they didn't ruin our lives.

Not the GP, but I was in a similar situation. It was a requirement to be able to get to the office if the situation required the lab to diagnose the problem.

Re: Breaking Up with On-Call

#99

Post by pjmlp »

zild3d wrote: post #69
rwmj wrote: post #64

Earlier quoted context omitted.

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half

So for say a weekly on-call rotation you would be paid all 24 hours x 7 days at double rate? (- ~40) Also most tech companies don't have unions...

In Europe most of the unions apply to the whole industry sector, what you do inside the building doesn't really matter.

Re: Breaking Up with On-Call

#100

Post by pjmlp »

gosub100 wrote: post #90
rwmj wrote: post #64

Earlier quoted context omitted.

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT 's job, whereas I'm SE . No thanks

That isn't how unions work, rather the usual US fearmongering of having them.
Post reply on HN