Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your company's on-call rotation like?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Ask HN: What is your company's on-call rotation like?

#51

Post by toomuchtodo »

RankingMember wrote: post #37
poooogles wrote: post #19

Earlier quoted context omitted.

>Oh no... just no. Production is not where debugging and testing should ever happen. Debugging sure. Testing in production however is incredibly valuable.

Are you kidding? Testing in a copy of production (integration, staging), sure. You should never be testing anything in production if you value your clients and reputation (and sanity).

That's what a canary is for, where you only send it a percentage of traffic.

I guarantee you that you'll never be able to accurately reproduce production traffic in staging or development.

Re: Ask HN: What is your company's on-call rotation like?

#52

Post by toomuchtodo »

tluyben2 wrote: post #26
throwawayoncall wrote: post #7

Work at a well funded Series B startup On-call is unpaid and no additional bump if called out or public holidays. No additional vacation in lieu either :( - Expected duties are to fix the issue and bring production systems back up, all engineers have context on production systems and can escalate if any additional context is needed. We have an established workflow of steps to take to determine root cause. We can also…

I would not consider working like that; 10-15 min response time? If on-call is only during office time then there is something about it although 10-15 minutes is mental. But outside office hours you need to get paid and paid a lot. Why would you do that unpaid? During the xmas period? You are getting screwed here.

> 10-15 min response time?

Unfortunately, yes. I'm (devops/sysadmin) in the same boat. Below market pay, but I work from home. It works for now.

Re: Ask HN: What is your company's on-call rotation like?

#54

Post by angelbob »

biot wrote: post #38
milkcircle wrote: post #35

Earlier quoted context omitted.

Appears to be less than 3k/day on average. Source: http://www.acep.org/Clinical---Practice-Management/Survey--M...

Thanks for looking this up. I did some research many years ago when I was in an unpleasant on-call situation (unreliable third-party dependency) and negotiated pay for it as a result. I recall seeing a $3500 daily rate figure at the time for a cardio surgeon, but that may have been an outlier. While the numbers may be off, one point I didn't make in my initial post but should have is that anybody doing on-call is pro…

Yeah, the article linked there is nicely specific. From the article (and these are median):

Neurological surgeons had the highest median daily rate for providing on-call coverage, about $2,000 a day. Near the top of the pay scale were neurologists ($1,500), cardiovascular surgeons ($1,600), internists ($1,050), and anesthesiologists ($800).

Among the specialists earning lower median daily rates for on-call compensation were: psychiatry ($500), general surgery ($500) gastroenterology ($500), ophthalmology ($300), and family medicine without obstetrics ($300), according to the MGMA survey data.

Re: Ask HN: What is your company's on-call rotation like?

#55

Post by angelbob »

acveilleux wrote: post #41
notmeknees wrote: post #16

Funny that I decided to finally create an account to answer this. Our team is heavily silo'ed, and each developer is responsible for a specific set of code. Since we have gone through a bunch of reductions and reorganizations, this ends up meaning that often one person is responsible. Our customer is a 24/7/365 operational unit, and it is our only customer. So, the end result is any given developer is on call all the…

Holy crap, I thought places like that only existed as straw mens in articles about bad practices. It'd unlikely that any other place would be worse, you have little to lose by changing jobs. A company that behaves like that is actively trying to find ways to cut your job too and they will just as soon as they think they can get away with it... Or maybe they'll cut someone else and have you do their job too, for effic…

The really good and really bad places tend to exist and not get talked about. That is, a surprising number of "strawman" workplaces, good and bad, tend to actually exist.

Re: Ask HN: What is your company's on-call rotation like?

#56

Post by bcbrown »

herval wrote: post #29
jeletonskelly wrote: post #14

To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call. Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2…

honest question: have you ever managed to work on a company that's either successful or growing very fast and can guarantee "various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2am call"? I can imagine that being possible in consultancies or small scale/load products, but honestly never seen it on anything larger than that - including environments with a mi…

I've worked at a late-stage startup that serves billions of requests per day at peak load, and there was no on-call for developers. I believe the ops team did have an on-call, but that was more at the infrastructure level, as everything was self-hosted at colos.

This was done by having a simple and resilient serving architecture. Every server was stateless. In addition, all the complicated logic was pre-computed into immutable lookup tables, offline. So if that task fails, it doesn't cause downtime, and can wait until the next workday.

We had a robust QA process, but it was far from stultifying.

Re: Ask HN: What is your company's on-call rotation like?

#57

Post by bcbrown »

nrr wrote: post #25
jeletonskelly wrote: post #14

To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call. Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2…

… whereas I'd conversely argue that volumes are also spoken when a company does not require its developers to be on-call. Often, those volumes are written with negative undertones and a narrative that speaks to developers not owning or understanding their software stack. Worse, in an exercise reminiscent of companies offering unlimited PTO, "not on-call," when read between the lines, could really mean "always on-call…

> Worse, in an exercise reminiscent of companies offering unlimited PTO, "not on-call," when read between the lines, could really mean "always on-call." I currently work with teams that operate this way, and the amount of burnout is staggering.

I've found the best way to screen for those sorts of problems is to ask everyone in the interview about their personal habits, instead of about process. Don't ask about vacation policy; ask when's the last time your interviewer went on vacation. Don't ask about the on-call rotation, ask about what issues came up outside work hours in the last month.

Re: Ask HN: What is your company's on-call rotation like?

#59

Post by laumars »

toomuchtodo wrote: post #53
laumars wrote: post #47

I'm basically expected to be 24/7. Which gets a bit much some weeks.

What stops you from looking for another job?

Nothing. :)

But for all the long hours and high stress levels, I do still quite enjoy my job. Which makes it all the more harder sometimes, because as bad as my current place might get on occasions, I know there's very few positions out there that are sufficiently better to encourage me to leave.

But the plus side is that it puts me in the fortunate position where I can turn down jobs. A luxury many people don't have.

Re: Ask HN: What is your company's on-call rotation like?

#60

Post by toomuchtodo »

laumars wrote: post #59
toomuchtodo wrote: post #53

Earlier quoted context omitted.

What stops you from looking for another job?

Nothing. :) But for all the long hours and high stress levels, I do still quite enjoy my job. Which makes it all the more harder sometimes, because as bad as my current place might get on occasions, I know there's very few positions out there that are sufficiently better to encourage me to leave. But the plus side is that it puts me in the fortunate position where I can turn down jobs. A luxury many people don't have…

Completely understand. I'm in the same position. Thanks for replying!
Post reply on HN