Live data from Hacker News

Being on call sucks

bobbiechen.com

191–200 of 282 posts

Re: Being on call sucks

#191

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

Another side note, doing a 4-day work week for engineering is a great and productivity boosting practice, but it shouldn't be seen as a carte blanche to not compensate them for after-hours emergencies. If you have a 4-day work week, then 32 hours a week is their normal full-time number of hours that you are compensating them for. When you ask them to go beyond this, you need to compensate them for their time.

Re: Being on call sucks

#192
post #179

Earlier quoted context omitted.

> What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. I don't think being available for work if an unlikely emergency comes up usually justifies the same comp as actively working that entire time.

Many orgs do comp their employees for the entire time they are on call. DoD does this for on-call employees due to past labor law issues, so it probably is legally required but rarely enforced.

DoD comps at 10% of overtime rate when not actively working an issue. https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/14...

The comment I replied to appeared to advocate for 100%.

Re: Being on call sucks

#193
post #25
post #19

Earlier quoted context omitted.

Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…

> Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second. And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep. It's happened to me, I felt awfully guilty fo…

>still the guilt made me feel like a failure and increased my anxiety when I'm on-call.

I don't know if this will help with your mindset at all, but as someone who's been listed as the secondary if the first responder doesn't answer: I really wouldn't beat yourself up over it. Self-correction is fine, but try not to let it drag you down too far. We have the same worries too. Our worries are usually "Crud. If I don't pick up, then it's either going to (big boss) or the customer is going to be firing off nasty emails in the morning and dragging me into some stupid meeting." The cause of why we're being paged doesn't really come into play. At most, I might just send a text or whatever to the first responder, to make sure that they're okay. "What if they aren't responding because they were in a car accident?"

If you have a good manager - and it sounds like you do, since you weren't reprimanded for performing a fundemental human function - then they are probably doing their best to look out for you and your interests. When people talk about "not being the boss" or "working as a team" or whatever, generally what they're trying to get at is that "They have your back". They've been in your position and have probably felt that same crush of emotion and worry that you might deal with. They are meant to be the final filter, in a hierarchy of filters, to protect you from those outside elements which ruin your ability to work effectively.

If you put in a sincere effort at your job, you have nothing to feel guilty over. That might be easier said than done, but it really is true. It just might take some time and practice to be able to forgive yourself when you make a mistake.

And if you feel like you could be doing a better job at this, or at that, then you just need to permit yourself the time to improve on that thing, and to remind yourself that you are allowed to mess up from time to time. It's about how you learn from those things.

Re: Being on call sucks

#194

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and…

The trick is to make sure the heroic efforts are rewarded quietly, but the unheroic ones create lots of noise.

This is inherently difficult because the heroic efforts tend to come as a result of very noisy incidents - everyone already knows the database cluster was literally on fire and wants to hear how Jane put it out while simultaneously remagnetizing the backup drives. The lead's job is to make the noise ensuring everyone knows Morgan's the one who did the boring rewiring of the backup-restore process to automatically sync and fail over to another AZ when thermal metrics start trending bad.

One thing I've noticed as we've been trying to integrate some on-call processes across formerly separate teams is that "our" incident reports were about 25% timeline/responses/impact analysis and 75% RCA and plans for future mitigations. "Theirs" were the opposite. Our incident rate has trended down sharply over the past two years even as our system grew; theirs has scaled up roughly at the same rate as their service count.

Re: Being on call sucks

#195

Earlier quoted context omitted.

Well in finance/trading, you need 24/5 or even 24/7 human monitoring. If you don't respond to that alert, you can wake up and see that the system just has lost tens of millions. Core engineers usually take on call duties, as fast diagnostics and response is critical.

What I have seen in finance/trading is that the required 24/7 monitoring is done by staffing 24/7 in pre-arranged shifts, not having this done by people being on-call while sleeping between their regular 9-5 workdays.

Yes, unless it is a problem that needs to be escalated to a core developer (which can always happen -- thankfully, rarely enough).

Re: Being on call sucks

#196

I enjoy being on call. I have heavy ADHD, so I am not a fan of routine, well-defined time activities and deadlines, but if I get this call at 2AM, I feel intense adrenaline rush, and I feel like a hero of the night saving the prod.

I'd like to learn more about that, it seems relevant to my situation. Some context (I'm not implying that this applies to you): we have an (well above average) capable engineer with a hero complex. In the most difficult of situations, he'll dive in and save the day (or night) where everyone else fails. He's also the one in the team who resisted standardisation and automation the most which, in a way, lead in the past…

Yes, this looks like my case. This is why I had to quit the corporate workforce and start working as a consultant — it provides me enough novelty and suits well to my strengths. I realize that in a regular corporate environment, I can be a liability instead.

Re: Being on call sucks

#197

Getting woken up in the middle of the night by a blasting noise is traumatic. I still have PTSD from my on call time. I swear that to subject a prisoner to that type of condition, where they are woken up at arbitrary times and forced to solve complex problems, would be considered cruel and unusual and inhumane treatment.

Sleep deprivation is indeed considered a form of psychological torture [1].

[1] https://pubmed.ncbi.nlm.nih.gov/31670701/

Re: Being on call sucks

#198
post #179

Earlier quoted context omitted.

What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. There should be no incentive to require or avoid "free" work, only advance planning of who'll be on call (with more available engineers doing more turns).

> What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. I don't think being available for work if an unlikely emergency comes up usually justifies the same comp as actively working that entire time.

I do. It means that time is not mine to do with as I like. I cannot spontaneously go out of town, nor can I decide to have a few drinks. I cannot start a project that needs to see completion - this includes simply involved dinners, hobby projects, and sometimes home repairs and car maintenance. If it is at night, I have to sacrifice sleep quality for being able to hear the phone.

In other words, they are working hours with some degree of personal freedom during the day. The assignment is to be ready in case things go wrong.

Re: Being on call sucks

#199
post #25
post #19

Earlier quoted context omitted.

Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…

> Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second. And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep. It's happened to me, I felt awfully guilty fo…

As someone who is second/third-tier on-call for most of the year: You know what's worse than getting paged when first-tier doesn't handle it? Not getting paged.

The only person I ever had to formally reprimand for on-call policy wasn't the one who failed to answer on time, it was the person who, three times, acked and didn't say or do anything else only because they didn't know how to fix it. In all cases I ended up getting pinged by someone else (head of our support team, or a colleague working an odd schedule), after the situation significantly worsened.

Higher tiers are there for a reason; don't feel bad if you've used them, only if you're expecting to use them.

(On the other hand, feeling guilty for making a mistake is also human, and to some degree something that makes me want such a person on my on-call team. People who feel better when they do a better job will do a better job!)

Re: Being on call sucks

#200
I lead an oncall rotation. It's important to stay on top of the pages/alerts, or else the rotation will rot and enter a death spiral where you don't get any sleep.

Every page (particularly the nighttime ones) are root-caused by the team every week. Each page gets one of four things:

1. Fix the code to handle the situation.

2. Tune the alert to increase the signal - resulting in a more actionable page.

3. Re-route to a more appropriate team.

4. Remove the alert if it doesn't help us keep the systems running.

So many pages were "informational" that we couldn't action and didn't indicate a problem that needed to be dealt with. Many others were bugs that people knew about but hadn't worked on because they didn't know it was waking us up! :)

Now, we get our sleep and people are asking to join the rotation!

Paying people to take the pager does not help when the rot sets in, but it does help encourage people to pick up extra shifts.

Post reply on HN