Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…
Being on call sucks
191–200 of 282 posts
Re: Being on call sucks
#192Earlier quoted context omitted.
> What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. I don't think being available for work if an unlikely emergency comes up usually justifies the same comp as actively working that entire time.
Many orgs do comp their employees for the entire time they are on call. DoD does this for on-call employees due to past labor law issues, so it probably is legally required but rarely enforced.
The comment I replied to appeared to advocate for 100%.
Re: Being on call sucks
#193Earlier quoted context omitted.
Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…
> Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second. And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep. It's happened to me, I felt awfully guilty fo…
I don't know if this will help with your mindset at all, but as someone who's been listed as the secondary if the first responder doesn't answer: I really wouldn't beat yourself up over it. Self-correction is fine, but try not to let it drag you down too far. We have the same worries too. Our worries are usually "Crud. If I don't pick up, then it's either going to (big boss) or the customer is going to be firing off nasty emails in the morning and dragging me into some stupid meeting." The cause of why we're being paged doesn't really come into play. At most, I might just send a text or whatever to the first responder, to make sure that they're okay. "What if they aren't responding because they were in a car accident?"
If you have a good manager - and it sounds like you do, since you weren't reprimanded for performing a fundemental human function - then they are probably doing their best to look out for you and your interests. When people talk about "not being the boss" or "working as a team" or whatever, generally what they're trying to get at is that "They have your back". They've been in your position and have probably felt that same crush of emotion and worry that you might deal with. They are meant to be the final filter, in a hierarchy of filters, to protect you from those outside elements which ruin your ability to work effectively.
If you put in a sincere effort at your job, you have nothing to feel guilty over. That might be easier said than done, but it really is true. It just might take some time and practice to be able to forgive yourself when you make a mistake.
And if you feel like you could be doing a better job at this, or at that, then you just need to permit yourself the time to improve on that thing, and to remind yourself that you are allowed to mess up from time to time. It's about how you learn from those things.
Re: Being on call sucks
#194Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…
Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and…
This is inherently difficult because the heroic efforts tend to come as a result of very noisy incidents - everyone already knows the database cluster was literally on fire and wants to hear how Jane put it out while simultaneously remagnetizing the backup drives. The lead's job is to make the noise ensuring everyone knows Morgan's the one who did the boring rewiring of the backup-restore process to automatically sync and fail over to another AZ when thermal metrics start trending bad.
One thing I've noticed as we've been trying to integrate some on-call processes across formerly separate teams is that "our" incident reports were about 25% timeline/responses/impact analysis and 75% RCA and plans for future mitigations. "Theirs" were the opposite. Our incident rate has trended down sharply over the past two years even as our system grew; theirs has scaled up roughly at the same rate as their service count.
Re: Being on call sucks
#195Earlier quoted context omitted.
Well in finance/trading, you need 24/5 or even 24/7 human monitoring. If you don't respond to that alert, you can wake up and see that the system just has lost tens of millions. Core engineers usually take on call duties, as fast diagnostics and response is critical.
What I have seen in finance/trading is that the required 24/7 monitoring is done by staffing 24/7 in pre-arranged shifts, not having this done by people being on-call while sleeping between their regular 9-5 workdays.
Re: Being on call sucks
#196I enjoy being on call. I have heavy ADHD, so I am not a fan of routine, well-defined time activities and deadlines, but if I get this call at 2AM, I feel intense adrenaline rush, and I feel like a hero of the night saving the prod.
I'd like to learn more about that, it seems relevant to my situation. Some context (I'm not implying that this applies to you): we have an (well above average) capable engineer with a hero complex. In the most difficult of situations, he'll dive in and save the day (or night) where everyone else fails. He's also the one in the team who resisted standardisation and automation the most which, in a way, lead in the past…
Re: Being on call sucks
#197Getting woken up in the middle of the night by a blasting noise is traumatic. I still have PTSD from my on call time. I swear that to subject a prisoner to that type of condition, where they are woken up at arbitrary times and forced to solve complex problems, would be considered cruel and unusual and inhumane treatment.
Re: Being on call sucks
#198Earlier quoted context omitted.
What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. There should be no incentive to require or avoid "free" work, only advance planning of who'll be on call (with more available engineers doing more turns).
> What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. I don't think being available for work if an unlikely emergency comes up usually justifies the same comp as actively working that entire time.
In other words, they are working hours with some degree of personal freedom during the day. The assignment is to be ready in case things go wrong.
Re: Being on call sucks
#199Earlier quoted context omitted.
Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…
> Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second. And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep. It's happened to me, I felt awfully guilty fo…
The only person I ever had to formally reprimand for on-call policy wasn't the one who failed to answer on time, it was the person who, three times, acked and didn't say or do anything else only because they didn't know how to fix it. In all cases I ended up getting pinged by someone else (head of our support team, or a colleague working an odd schedule), after the situation significantly worsened.
Higher tiers are there for a reason; don't feel bad if you've used them, only if you're expecting to use them.
(On the other hand, feeling guilty for making a mistake is also human, and to some degree something that makes me want such a person on my on-call team. People who feel better when they do a better job will do a better job!)
Re: Being on call sucks
#200Every page (particularly the nighttime ones) are root-caused by the team every week. Each page gets one of four things:
1. Fix the code to handle the situation.
2. Tune the alert to increase the signal - resulting in a more actionable page.
3. Re-route to a more appropriate team.
4. Remove the alert if it doesn't help us keep the systems running.
So many pages were "informational" that we couldn't action and didn't indicate a problem that needed to be dealt with. Many others were bugs that people knew about but hadn't worked on because they didn't know it was waking us up! :)
Now, we get our sleep and people are asking to join the rotation!
Paying people to take the pager does not help when the rot sets in, but it does help encourage people to pick up extra shifts.