On-call is just fine as long as: 1. There is a rotation. 2. There are few pages, preferably the median should be 0 per week. 3. Spurious / non-actionable alerts get fixed right away (with very high priority) 4. You're not up more than 1 week per 1-1.5 month. 5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime. As wit…
Being on call sucks
91–100 of 282 posts
Re: Being on call sucks
#92After being a SWE for a decade in Europe, I have never heard of anyone in my network who needed to be on-call. Is this a US thing or only for devops? Why would a software engineer need to be on call ever? That just means the CICD/testing/validation pipeline sucks.
And there are plenty of cases where you need to fix something ASAP and the Ops team can't do it.
Re: Being on call sucks
#93After years of iteration, here’s what our team does. The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe. This gets us as close to round the world coverage as we can have. There are two people on call for each shift, each shift lasts a week. It will typically (but not always) be one person from US and one person from UK/EU. This helps reduce the single personal c…
What does that mean in practice?
Hopefully: "This woke me up last night. It's now the top priority until it's fixed so it never wakes anyone up again. Sorry product manager, your new feature will have to wait."
Re: Being on call sucks
#94So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong. At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid wh…
Re: Being on call sucks
#95I enjoy being on call. I have heavy ADHD, so I am not a fan of routine, well-defined time activities and deadlines, but if I get this call at 2AM, I feel intense adrenaline rush, and I feel like a hero of the night saving the prod.
Re: Being on call sucks
#96Earlier quoted context omitted.
You're using "never" everywhere here. That is in my opinion the main red flag here. On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds ha…
almost always there is somebody who's plan is "sitting at home" and nobody minds having the pager in such circumstances I wonder if this is an American attitude that exists primarily because we (collectively) have allowed our employers to demand this of us? Personally, I hate being on call. Most weekends, I spend at least 6 hours cycling. Sometimes significantly more. I go camping regularly. I go kayaking regularly.…
Re: Being on call sucks
#97Earlier quoted context omitted.
How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.
I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not…
Re: Being on call sucks
#98Re: Being on call sucks
#99So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong. At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid wh…
How does Google deal with issues caused by downstream systems causing alerts?
Re: Being on call sucks
#100When I worked as a full time DoD scientist, there was a very set-in-place system for dealing with these situations, and it was normal to pay double overtime for any hours spent by an employee on after hours emergencies. This is the right way to do it. Pre-series B startups largely don't do this, but once you get to series B and C suddenly it becomes a thing because companies realize they either have to legally, or have to to prevent their employees from churning and to protect themselves from people not showing up to put out the fire.
Just do it, and do it early. Do it before Series A. That's the advice I give my consulting clients, and the approach I take with my own companies. By compensating your employees for this time you also take what would be a red flag for many would-be employees and turn it into an exciting perk.