Live data from Hacker News

Being on call sucks

bobbiechen.com

91–100 of 282 posts

Re: Being on call sucks

#91
post #6

On-call is just fine as long as: 1. There is a rotation. 2. There are few pages, preferably the median should be 0 per week. 3. Spurious / non-actionable alerts get fixed right away (with very high priority) 4. You're not up more than 1 week per 1-1.5 month. 5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime. As wit…

n+1: The people that wrote the code and designed the systems are the first to be paged. Their work, their disrupted sleep.

Re: Being on call sucks

#92

After being a SWE for a decade in Europe, I have never heard of anyone in my network who needed to be on-call. Is this a US thing or only for devops? Why would a software engineer need to be on call ever? That just means the CICD/testing/validation pipeline sucks.

It's not US only. US companies also do it in Europe.

And there are plenty of cases where you need to fix something ASAP and the Ops team can't do it.

Re: Being on call sucks

#93
post #26

After years of iteration, here’s what our team does. The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe. This gets us as close to round the world coverage as we can have. There are two people on call for each shift, each shift lasts a week. It will typically (but not always) be one person from US and one person from UK/EU. This helps reduce the single personal c…

> Anything that is an annoyance during on-call is a candidate for review and change.

What does that mean in practice?

Hopefully: "This woke me up last night. It's now the top priority until it's fixed so it never wakes anyone up again. Sorry product manager, your new feature will have to wait."

Re: Being on call sucks

#94
post #29

So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong. At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid wh…

I had no idea you essentially get “overtime” pay for on call at google. That’s how it should be imo. I’ve avoided on call jobs for the lack of extra pay for doing more work.

Re: Being on call sucks

#95

I enjoy being on call. I have heavy ADHD, so I am not a fan of routine, well-defined time activities and deadlines, but if I get this call at 2AM, I feel intense adrenaline rush, and I feel like a hero of the night saving the prod.

I'd like to learn more about that, it seems relevant to my situation. Some context (I'm not implying that this applies to you): we have an (well above average) capable engineer with a hero complex. In the most difficult of situations, he'll dive in and save the day (or night) where everyone else fails. He's also the one in the team who resisted standardisation and automation the most which, in a way, lead in the past to situations where his unique skills were required.

Re: Being on call sucks

#96

Earlier quoted context omitted.

You're using "never" everywhere here. That is in my opinion the main red flag here. On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds ha…

almost always there is somebody who's plan is "sitting at home" and nobody minds having the pager in such circumstances I wonder if this is an American attitude that exists primarily because we (collectively) have allowed our employers to demand this of us? Personally, I hate being on call. Most weekends, I spend at least 6 hours cycling. Sometimes significantly more. I go camping regularly. I go kayaking regularly.…

I believe the point was more than if you have a team of, say, ten people, there's always someone who is not busy for some given night, and there's some reasonable trade that can be worked out, the moreso if oncall risk is considered low by the team. Obviously as you scale down that becomes less true.

Re: Being on call sucks

#97
post #45
post #39

Earlier quoted context omitted.

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not…

No post body was provided.

Re: Being on call sucks

#99
post #29

So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong. At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid wh…

> alerts caused by issues in downstream systems that you could do nothing about

How does Google deal with issues caused by downstream systems causing alerts?

Re: Being on call sucks

#100
Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of their regular hours without compensation. Another very important angle to consider is you want to incentivize your employees to answer the call to deal with these issues. If there is no incentive structure in place, they might not take it seriously, which in itself can be a security and stability risk that could threaten things like your SOC-2 compliance, if you have it.

When I worked as a full time DoD scientist, there was a very set-in-place system for dealing with these situations, and it was normal to pay double overtime for any hours spent by an employee on after hours emergencies. This is the right way to do it. Pre-series B startups largely don't do this, but once you get to series B and C suddenly it becomes a thing because companies realize they either have to legally, or have to to prevent their employees from churning and to protect themselves from people not showing up to put out the fire.

Just do it, and do it early. Do it before Series A. That's the advice I give my consulting clients, and the approach I take with my own companies. By compensating your employees for this time you also take what would be a red flag for many would-be employees and turn it into an exciting perk.

Post reply on HN