Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being on call sucks

bobbiechen.com

Post reply on HN
Print view View original

151–160 of 282 posts

Re: Being on call sucks

#151

Post by PeterisP »

monster_group wrote: post #51
jon-wood wrote: post #15

There are many issues with being on-call, particularly in environments where false alerts routinely happen, and where management aren't in the rota so don't directly feel the pain. One of my biggest though is the concept of a weekly rota, with people being on-call for a full week at a time. Sometimes that works fine, and you'll get no alerts all week, but incidents tend to cluster. If something has changed that cause…

Daily rotation seems too frequent to me. Sometimes you need 2-3 days to really deep dive and fix the core issue after the alarm has gone silent. Also, with daily rotation it is tempting to wait it out and punt the problem to the next person.

Fixing the core issue should be done during regular day hours on a normal work schedule, so it does not justify a longer on-call shift. If something needs 2-3 days of deep dive, you should get proper time for rest (i.e. not on-call) between those days.

Re: Being on call sucks

#152

Post by Throwiemcthrow »

Tallianar wrote: post #16

My company(UK) recently tried to force on-call on all engineers. The initial wording was very restrictive, like 5 minute acknowledgement time and 15 minutes at-laptop. 24/7 for 7 days. They tried to have this implemented without any extra remuneration or perks for the on-call engineer. On top of it possibly being very illegal, it seems very immoral to spring something like that on people that did on agree to it when…

I joined Prospect because my company tried to implement an unspoken on-call arrangement, whereby they would try to call me on my mobile 24/7 expecting an immediate response. I asked what the additional renumeration is for that, and they said there isn't any.

Now I'm a Prospect member, and my mobile is always on mute.

Re: Being on call sucks

#153

Post by raverbashing »

valeness wrote: post #143
bpodgursky wrote: post #116

Earlier quoted context omitted.

Unless you're in a really dysfunctional startup, the employees who voluntarily jump in after-hours to fix OC issues are going to be very quickly compensated with raises and refresher option grants. Far better payout generally than a few bucks of overtime. It becomes really obvious who is holding the company together and who is coasting, relying on the "senior" (ie, anyone who puts in the effort) engineers to keep the…

Lolol, I've been jumping in after-hours for years. They only take notice when you say "I quit".

And some even later than that, when they go to fix something and they realize they don't know how

Re: Being on call sucks

#154

Post by sam0x17 »

sokoloff wrote: post #125
sam0x17 wrote: post #100

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

What I find interesting is that if you give employees a choice of 100 units of currency per year as salary plus 1 unit of currency for each week on call with an expectation of 1 week on call every four weeks versus 115 units of currency for the year with the same on-call expectations, some employees will feel better about the first arrangement and some will feel better about the second arrangement. Some people want t…

The problem is once you consider it part of your standard comp, you take it for granted. If emergencies almost never happen and then one does, you will inevitably feel cheated when you have to wake up at 2 AM on a Sunday. Much better familial optics if you can tell your family "hey we got an extra $500 for that" versues "oh, dealing with these is included in my take-home salary".

Even worse, if emergencies happen all the time, your family is going to hate you if you don't have some reward you can show for each one, especially if there are other jobs at your salary range that don't have an on-call clause.

Re: Being on call sucks

#155

Post by Aeolun »

dkyc wrote: post #39
oogali wrote: post #26

After years of iteration, here’s what our team does. The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe. This gets us as close to round the world coverage as we can have. There are two people on call for each shift, each shift lasts a week. It will typically (but not always) be one person from US and one person from UK/EU. This helps reduce the single personal c…

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

If this happened when I was at the bar half drunk I’d always say something to the extend of ‘do you really want me to work on the system right now?’.

Sometimes the answer is yes. Sometimes the answer is no. If it’s down already it can’t get worse.

To be fair, I was the first (and only) point of escalation.

Re: Being on call sucks

#156

Post by PeterisP »

atemerev wrote: post #86
ricardobayes wrote: post #76

After being a SWE for a decade in Europe, I have never heard of anyone in my network who needed to be on-call. Is this a US thing or only for devops? Why would a software engineer need to be on call ever? That just means the CICD/testing/validation pipeline sucks.

Well in finance/trading, you need 24/5 or even 24/7 human monitoring. If you don't respond to that alert, you can wake up and see that the system just has lost tens of millions. Core engineers usually take on call duties, as fast diagnostics and response is critical.

What I have seen in finance/trading is that the required 24/7 monitoring is done by staffing 24/7 in pre-arranged shifts, not having this done by people being on-call while sleeping between their regular 9-5 workdays.

Re: Being on call sucks

#157

Post by Aeolun »

oofnik wrote: post #54

A major issue with on-call, and certainly one I've encountered multiple times, is the high likelihood of moral hazard - the people who are responsible for addressing incidents are not the same people who designed and maintained the system at fault. This results in the former team feeling powerless to put out fires which could have been prevented by more robust design, and the latter team having no incentive to improv…

> SRE gets this right, at least in theory, by requiring that all production systems be reviewed and approved, including observability and incident management procedures, prior to entering service.

Both doing, and being subjected to reviews suck balls. I’d much rather be on call 24/7 to fix the problems I caused myself.

Re: Being on call sucks

#158

Post by matsemann »

Broken_Hippo wrote: post #7
cube2222 wrote: post #6

On-call is just fine as long as: 1. There is a rotation. 2. There are few pages, preferably the median should be 0 per week. 3. Spurious / non-actionable alerts get fixed right away (with very high priority) 4. You're not up more than 1 week per 1-1.5 month. 5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime. As wit…

6. You are paid an amount for every hour of being on-call. This should be on top of your base pay. This is important. If you are on-call, you do not have the freedom that you would otherwise have and you should be compensated for this.

Also, only on on-call for things you're actually responsible for. I hate having on-calls span multiple teams. They can then be lazy, as their issues hurts someone else. And the added stress of having to debug and fix shit you're not comfortable touching.

Additionally, I'll never accept on-call with 15 minutes from alert to being on a computer again. It's just too limiting and disruptive of my life. Have to bring the computer everywhere. Any dinner or social event can be instantly ruined. A workout becomes meaningless. I remember doing a swim and having to check my phone every 5 minutes. It's mentally exhausting and frankly not worth the pay.

Re: Being on call sucks

#159

Post by bogomipz »

sethammons wrote: post #77

My last gig, on call worked well, I thought, for a few reasons: it was our services that we wrote, it was 1 wk out of 6 that you were on call, we heavily prioritized fixing unactionable alerts and automating fixes -- every alert had a runbook entry that described the non-automated fixes, and while on call your or sprint commitments were not counted, That last point was very nice as it meant you could work on whatever…

>"My last gig, on call worked well, I thought, for a few reasons: it was our services that we wrote, it was 1 wk out of 6 that you were on call, ..."

Am I the only one that thinks the prospect being on call one week every six is horrifying?

Re: Being on call sucks

#160

Post by valenterry »

kansface wrote: post #148
valenterry wrote: post #147

Why not just have on-call being an auction? Some people hate on call, some people love it since they are usually at home playing games anyways or don't mind so much being disturbed during sleep and love to make some extra money on the side. So just auction the on-call times and have employees bid for them. Naturally, bids for on-call at christmas will probably be lower and for some other times higher. Employer can se…

That would severely misalign the incentives of the two groups which leads to trouble over time.

How so?
Post reply on HN