Live data from Hacker News

Ask HN: What is your company's on-call rotation like?

news.ycombinator.com

31–40 of 64 posts

Re: Ask HN: What is your company's on-call rotation like?

#31
post #11

For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line shou…

3k a day? do you have a source for that?

Re: Ask HN: What is your company's on-call rotation like?

#32
Programmer at a Hospital -- Only have to do stuff if I get paged (rotation is 1 every 4 weeks for 7 days, 5PM-8AM). If the issue(s) affects a clinical caregiver then the issue needs to be resolved asap no matter what, priority is only regarding the issue you were paged for, if the on-call person can't resolve they will usually page additional people or escalate to a manager who can find the right people. If another team's issues are causing us problems we just need to inform the customer about what's going on and then keep them updated.

Re: Ask HN: What is your company's on-call rotation like?

#33
post #25

To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call. Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2…

… whereas I'd conversely argue that volumes are also spoken when a company does not require its developers to be on-call. Often, those volumes are written with negative undertones and a narrative that speaks to developers not owning or understanding their software stack. Worse, in an exercise reminiscent of companies offering unlimited PTO, "not on-call," when read between the lines, could really mean "always on-call…

This. tl;dr: No on-call schedule sometimes means always on-call.

Re: Ask HN: What is your company's on-call rotation like?

#34
post #11

For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line shou…

They probably get a lot of repeat work when they tell the patient how much it's costing them.

Re: Ask HN: What is your company's on-call rotation like?

#35
post #31
post #11

For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line shou…

3k a day? do you have a source for that?

Appears to be less than 3k/day on average. Source: http://www.acep.org/Clinical---Practice-Management/Survey--M...

Re: Ask HN: What is your company's on-call rotation like?

#36
1) We don't keep a formal on call but 3 of us are tied to an automated alert system and whoever has a chance to take care of it, does. We are all full stack devs so generally we can fix it at the time. If its complicated, we can help it hobble and fix it later.

2) We get 3-4 alerts a year that have to be handled before the next business day.

3) As such, there is no real work priority, triage, etc. You resolve it immediately. There is no other priority. [ Any on-call event == lost money ]

> * how do you manage for other teams' risk? (ie their api goes down, you can't satisfy your customers)

Asynchronous processing. I buffer until their API is back up. I do this for literally dozens of companies from small manufacturers to Amazon.

There really isn't any other good way to handle it and if you need to do it otherwise that really is a fundamental architectural problem that should have been resolved at the design phase.

Re: Ask HN: What is your company's on-call rotation like?

#37

Earlier quoted context omitted.

> So, get a call at 2 AM on the weekend and design and implement a new feature, release to production to debug and test. Oh no... just no. Production is not where debugging and testing should ever happen, obviously. If that's what your company expects I'm not surprised QA was let go when your customers are doing it for you. This makes me sad.

>Oh no... just no. Production is not where debugging and testing should ever happen. Debugging sure. Testing in production however is incredibly valuable.

Are you kidding? Testing in a copy of production (integration, staging), sure. You should never be testing anything in production if you value your clients and reputation (and sanity).

Re: Ask HN: What is your company's on-call rotation like?

#38
post #31

Earlier quoted context omitted.

3k a day? do you have a source for that?

Appears to be less than 3k/day on average. Source: http://www.acep.org/Clinical---Practice-Management/Survey--M...

Thanks for looking this up. I did some research many years ago when I was in an unpleasant on-call situation (unreliable third-party dependency) and negotiated pay for it as a result. I recall seeing a $3500 daily rate figure at the time for a cardio surgeon, but that may have been an outlier.

While the numbers may be off, one point I didn't make in my initial post but should have is that anybody doing on-call is providing a valuable service. It's no different than paying a babysitter to watch TV while your kids sleep. Both babysitters and devops/IT/whatever are compensated for their ability to act in case the unexpected happens. If someone's not getting paid for that, then it must not be valuable to the company to have that coverage.

Re: Ask HN: What is your company's on-call rotation like?

#39
post #25

To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call. Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2…

… whereas I'd conversely argue that volumes are also spoken when a company does not require its developers to be on-call. Often, those volumes are written with negative undertones and a narrative that speaks to developers not owning or understanding their software stack. Worse, in an exercise reminiscent of companies offering unlimited PTO, "not on-call," when read between the lines, could really mean "always on-call…

Your points are valid. Stronger than that, I stand more to your view than the parents. Although I certainly understand the desire of the parent, I've been on both the Ops and Dev side of that fence, and on call REALLY SUCKS (I almost feel dirty in this post agreeing with a statement of "I should be on call" but it is what it is :) ) but it sucks worse if as you say it's entirely on the backs of the ops who are also handling sysadmin style work, and are kept more isolated from dev as is typical in "throw it over the wall" shops. (In writing this I realize I want to clarify that I have other thoughts entirely about having ops specialists instead of suggesting a merge to totally unified devops, but that's an entirely other discussion). Also as you say, ownership/responsibility/etc is also an ancillary benefit.

What I actually wanted to say from all this however, is that despite the truth of your argument, I'd take a third angle. No on call is bad, but similarly, _unpaid_ OT is bad. The common rhetoric from this soapbox is that if employers are accountable for this time, it'll incentivise them to have processes and hold values that don't abuse an expensive resource. If something is on fire; the on call employee still _will be there to handle it_, but the fact that it's become a pro-bono assumed part of many of our jobs in the tech sector is the part I take issue with more than doing the task itself.

Re: Ask HN: What is your company's on-call rotation like?

#40
post #11

For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line shou…

>Ensure you are compensated appropriately.

I agree 100%

My last company is notorious for official-unofficial on call duties.

You must remain within 30 minutes of the office, be near your cell phone and sober, 24x7. Because there is no official policy, there was no rotation, so technically I was always on call. Their official stance on compensation is that salaried employees can't get overtime, and it's covered by the annual bonus anyway. Yep, unlimited on-call & overtime covered by a ~7% bonus that pays out about 50% of the time.

I was once out of the country on a statutory holiday and didn't answer my personal cell phone (roaming would have killed me), which caused me to get reprimanded and lose my bonus for the entire year.

After some back and forward I quit soon after.

Post reply on HN