Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being on call sucks

bobbiechen.com

Post reply on HN
Print view View original

51–60 of 282 posts

Re: Being on call sucks

#51

Post by monster_group »

jon-wood wrote: post #15

There are many issues with being on-call, particularly in environments where false alerts routinely happen, and where management aren't in the rota so don't directly feel the pain. One of my biggest though is the concept of a weekly rota, with people being on-call for a full week at a time. Sometimes that works fine, and you'll get no alerts all week, but incidents tend to cluster. If something has changed that cause…

Daily rotation seems too frequent to me. Sometimes you need 2-3 days to really deep dive and fix the core issue after the alarm has gone silent. Also, with daily rotation it is tempting to wait it out and punt the problem to the next person.

Re: Being on call sucks

#52

Post by tartoran »

__alexs wrote: post #30
Broken_Hippo wrote: post #7

Earlier quoted context omitted.

6. You are paid an amount for every hour of being on-call. This should be on top of your base pay. This is important. If you are on-call, you do not have the freedom that you would otherwise have and you should be compensated for this.

This is also an excellent way to incentivise the business to actually build resilient systems that minimise the need for on-call hours.

Yes! On call should be expensive for the company so over time the systems become stronger.

Re: Being on call sucks

#53

Post by Sodman »

cube2222 wrote: post #38
nickjj wrote: post #19

Earlier quoted context omitted.

Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…

You're using "never" everywhere here. That is in my opinion the main red flag here. On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds ha…

The company having that much influence over your day-to-day life during non-business hours every fourth week is an enormous burden. I would want at least 50% higher overall total annual comp to even consider that schedule, and even then it's still only a maybe.

Re: Being on call sucks

#54

Post by oofnik »

A major issue with on-call, and certainly one I've encountered multiple times, is the high likelihood of moral hazard - the people who are responsible for addressing incidents are not the same people who designed and maintained the system at fault. This results in the former team feeling powerless to put out fires which could have been prevented by more robust design, and the latter team having no incentive to improve reliability.

SRE gets this right, at least in theory, by requiring that all production systems be reviewed and approved, including observability and incident management procedures, prior to entering service. This ensures that there is some shared responsibility across teams for maintaining uptime.

https://sre.google/sre-book/being-on-call/

Re: Being on call sucks

#55

Post by di4na »

The one weird trick is to pay people to fix it. There is an incentive problem here.

The second one weird trick is to legitimately ask people about the on call experience. Again. And again. And again.

The reason nothing change is because noone is incentivised to change it.

Re: Being on call sucks

#56

Post by oogali »

dkyc wrote: post #39
oogali wrote: post #26

After years of iteration, here’s what our team does. The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe. This gets us as close to round the world coverage as we can have. There are two people on call for each shift, each shift lasts a week. It will typically (but not always) be one person from US and one person from UK/EU. This helps reduce the single personal c…

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

Even in all the situations you’ve listed, I still have my phone with me.

If I’m going to be out of cell coverage (e.g. a plane ride, or in the countryside with spotty Internet) or simply want to be left alone, I usually plan for that in advance and do a combination of: 1) scaling back our risk exposure by rescheduling work (which requires you to have a good understanding of the business, its needs, and its timelines) and 2) shoring up the bits I feel most weary about through code, documentation, tooling, and/or contractors.

The same goes for the team SMEs: reschedule where I can, crosstrain where I can’t, get headcount where none of the prior work.

I’ve been on call since 1999. I had to figure out a pattern that worked for me (and my family) but wouldn’t result in a life that was boring or worse, one that I resented.

Re: Being on call sucks

#57

Post by thesuperbigfrog »

oofnik wrote: post #54

A major issue with on-call, and certainly one I've encountered multiple times, is the high likelihood of moral hazard - the people who are responsible for addressing incidents are not the same people who designed and maintained the system at fault. This results in the former team feeling powerless to put out fires which could have been prevented by more robust design, and the latter team having no incentive to improv…

>> A major issue with on-call, and certainly one I've encountered multiple times, is the high likelihood of moral hazard - the people who are responsible for addressing incidents are not the same people who designed and maintained the system at fault. This results in the former team feeling powerless to put out fires which could have been prevented by more robust design, and the latter team having no incentive to improve reliability.

The Amazon approach was (still is?) to have the team that develops and deploys the software to be responsible for the on-call rotation for that system.

If you are developing software for such a team, it gives you a direct reason to make sure everything is designed and tested well before it is deployed to production--you (or a teammate) will be answering the early morning alert to fix it if it is not.

A direct feedback loop like that is remarkably effective to prevent the moral hazard and ensures direct accountability when buggy software gets deployed.

Re: Being on call sucks

#58

Post by quickthrower2 »

square_usual wrote: post #48
coldcode wrote: post #43

I worked at a place in the 2000's where our main application leaked like a sieve due to not releasing memory from a C++ framework, and two people had to take turns every other night restarting the app (actually it was chopped up into 20 different apps that had to be started in order by hand) every two hours or so. I can't imagine they got much sleep on their night.

That seems like a prime candidate for scripting.

And monolith-ing

Re: Being on call sucks

#59

Post by throwaddzuzxd »

cube2222 wrote: post #8
Broken_Hippo wrote: post #7

Earlier quoted context omitted.

6. You are paid an amount for every hour of being on-call. This should be on top of your base pay. This is important. If you are on-call, you do not have the freedom that you would otherwise have and you should be compensated for this.

If it's in the contract you sign, then it's already priced in. I don't see much value in specifically outlining which part is base and which is for oncall, if oncall is mandatory.

French here.

My contract says that if on-calls are needed, I might have to be in the rotation. This clause increases my pay rate even if I'm not on-call.

If however I am actually on-call, I am paid more. And if the on-call rings, I'm again paid more on top of the on-call period. And as the French law mandates 11 consecutive hours of rest, if the on-call rings in the middle of the night, I'll usually come to work later the day after.

If you don't have advantages for being on-call, you're the one being taken advantage of.

Re: Being on call sucks

#60

Post by formerkrogemp »

nickjj wrote: post #19
stevekemp wrote: post #13

Earlier quoted context omitted.

The Finnish collective agreement, which covers IT workers, spells this out explicitly: * For every hour you're available for on-call you get your regular hourly rate. * For every incident, which involves you working, you receive twice your hourly salary for the duration. So companies based her tend to have that as standard, though I'm sure some companies would pay more to stand out.

Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…

> Finally, somewhere that I fully agree with.

I think we could all stand to learn a bit from the Finnish and the Norwegians.

Post reply on HN