Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being on call sucks

bobbiechen.com

Post reply on HN
Print view View original

211–220 of 282 posts

Re: Being on call sucks

#211

Post by hyperman1 »

Aeolun wrote: post #138

> Does anyone have One Weird Trick™ to fix it? My one weird trick is to have a zero tolerance policy for flaky monitors/tests. If it’s not accurate, we either have to drop everything else and fix it, or disable that alarm entirely. Like they say, normalization of deviance is real, and the only way to fight against it is to have every form of deviance be a problem.

I tend to agree with you, but found an exception a while ago.

A certain file has to appear before a specific time, or else some people don't get money they deserve and rightfully get very angry.

Except 1 or 2 times per year there is nobody in that situation. No payments have to be made. No file appears, as other alerts would signal an empty file.

As the relevant time was in business hours and the thing was important, I decided to swallow my pride and accept that invalid alert.

  Resolution procedure is documented as: Call team X, and ask if this is correct.  If yes, blackout that alert for 24 hours

Re: Being on call sucks

#212

Post by wonderwonder »

Favorite part of quitting my last nightmare of a job was that I continued to get text notifications that I was the on call person for the night/weekend for ~6 months after I left. I slept well each night knowing I had the something is broken notification number blocked

Re: Being on call sucks

#213

Post by xur17 »

niij wrote: post #184
PeterisP wrote: post #136

Earlier quoted context omitted.

IMHO the whole notion of "week of on-call" is ridiculous. Being on-call ready is a shift of (hopefully low-intensity) work time. You can do 8 hour shifts, 12 hour shifts, 24 hour shifts, but you can't do 168 hour shifts for monitoring something, anything - that's impractical and should be illegal. There are (or should be) some minimum standards of rest time that every human being must get, and being on-call is not it…

On-call is meant to be for rare emergencies. If you're getting paged outside of work hours on each 1 week shift then something is wrong. It should in no way be equivalent to 168 hours of work.

Agreed. I feel like there are significantly different "on call" expectations being discussed in the comments for this post.

Re: Being on call sucks

#214

Post by maerF0x0 »

scottlamb wrote: post #189
sam0x17 wrote: post #100

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and…

> They're often avoidable.

Avoidance isn't the goal of mgmt though. Cost minimization is, and by quantizing the compensation portion it can help figure out the cost of a bug. Ultimately the cost of a bug is a key variable in the decision to prevent or solve.

The cost can easily vary from $0 (eg at a startup with no customers) or to millions (maybe billions?) when you consider the risk of brand damage, lost ARR (multiply that ARR by the sales multiple!), decremented velocity (lost opportunity value), and eventually even recruiting cost (to replace frustrated employees).

I've always wanted to work at a JoelTest[1] company which fixes bugs before writing new code... But the closest i've found is a commitment that the oncall engineer works on bugs for their rotation to reduce the bug backlog ( and is not part of sprint velocity)

[1]: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

Re: Being on call sucks

#215

Post by akhmatova »

wonderwonder wrote: post #212

Favorite part of quitting my last nightmare of a job was that I continued to get text notifications that I was the on call person for the night/weekend for ~6 months after I left. I slept well each night knowing I had the something is broken notification number blocked

Yup - those fleshpots where the code base is too convoluted, or everyone is "too busy" to tend to such matters as who gets notified for what (and to, you know, update these settings once people leave the org) probably have lots of other red flags. So you can definitely sleep better knowing you managed to escape that one.

Re: Being on call sucks

#216

Post by xur17 »

Aeolun wrote: post #155
dkyc wrote: post #39

Earlier quoted context omitted.

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

If this happened when I was at the bar half drunk I’d always say something to the extend of ‘do you really want me to work on the system right now?’. Sometimes the answer is yes. Sometimes the answer is no. If it’s down already it can’t get worse. To be fair, I was the first (and only) point of escalation.

> If it’s down already it can’t get worse.

Oh yes it can.

Re: Being on call sucks

#217

Post by scottlamb »

maerF0x0 wrote: post #214
scottlamb wrote: post #189

Earlier quoted context omitted.

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and…

> They're often avoidable. Avoidance isn't the goal of mgmt though. Cost minimization is, and by quantizing the compensation portion it can help figure out the cost of a bug. Ultimately the cost of a bug is a key variable in the decision to prevent or solve. The cost can easily vary from $0 (eg at a startup with no customers) or to millions (maybe billions?) when you consider the risk of brand damage, lost ARR (multi…

> Avoidance isn't the goal of mgmt though.

Management's goals are easier to achieve when they can employ excellent engineers. As I explained, I don't want to work somewhere heroics are often needed. I'm not alone. That's a factor they need to consider, in addition to the customer impact of the outages themselves.

Re: Being on call sucks

#218

Post by sam0x17 »

maerF0x0 wrote: post #214
scottlamb wrote: post #189

Earlier quoted context omitted.

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and…

> They're often avoidable. Avoidance isn't the goal of mgmt though. Cost minimization is, and by quantizing the compensation portion it can help figure out the cost of a bug. Ultimately the cost of a bug is a key variable in the decision to prevent or solve. The cost can easily vary from $0 (eg at a startup with no customers) or to millions (maybe billions?) when you consider the risk of brand damage, lost ARR (multi…

Even at a startup with zero customers the cost of a bug is non-zero if you plan to allocate dev time to it or you have to fix it to be able to launch.

Re: Being on call sucks

#219

Post by Tao3300 »

Aeolun wrote: post #138

> Does anyone have One Weird Trick™ to fix it? My one weird trick is to have a zero tolerance policy for flaky monitors/tests. If it’s not accurate, we either have to drop everything else and fix it, or disable that alarm entirely. Like they say, normalization of deviance is real, and the only way to fight against it is to have every form of deviance be a problem.

99% of the time your shit just isn't that essential. That's the One Weird Trick: don't get suckered into thinking your corporate vision is so important that it can't have an issue wait until morning.

Re: Being on call sucks

#220

Post by henryfjordan »

kayodelycaon wrote: post #70

On-call is even worse for people with disabilities. I quite literally can't do it unless I stop taking my antipsychotic. Under ADA, I can not be placed on call, regardless of policy, nor can I be discriminated against for that. On-call is not an essential function of being a software developer, with very few exceptions—all of which have nothing to do with "policy" or "fairness". Needless to say, companies (and some c…

Unless you've tested this theory in court it might not be true. It's almost certainly not as cut and dry as you make it seem. Many companies put the same people oncall who write the code, meaning it literally is an essential function of a software developer to provide oncall support. You'd have to argue in court that it's not really essential but it would be situationally dependent.

That said, I'd hope most places would be willing to accommodate you. Places I've worked have always treated oncall as a kinda optional "right thing to do". I've never seen anyone punished for missing an alert. You'd have a good argument if that were the case at your company but that approach to oncall is not universal.

Post reply on HN