Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your company's on-call rotation like?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Ask HN: What is your company's on-call rotation like?

#41

Post by acveilleux »

notmeknees wrote: post #16

Funny that I decided to finally create an account to answer this. Our team is heavily silo'ed, and each developer is responsible for a specific set of code. Since we have gone through a bunch of reductions and reorganizations, this ends up meaning that often one person is responsible. Our customer is a 24/7/365 operational unit, and it is our only customer. So, the end result is any given developer is on call all the…

Holy crap, I thought places like that only existed as straw mens in articles about bad practices. It'd unlikely that any other place would be worse, you have little to lose by changing jobs.

A company that behaves like that is actively trying to find ways to cut your job too and they will just as soon as they think they can get away with it... Or maybe they'll cut someone else and have you do their job too, for efficiency.

Re: Ask HN: What is your company's on-call rotation like?

#42

Post by ChrisArgyle »

notmeknees wrote: post #16

Funny that I decided to finally create an account to answer this. Our team is heavily silo'ed, and each developer is responsible for a specific set of code. Since we have gone through a bunch of reductions and reorganizations, this ends up meaning that often one person is responsible. Our customer is a 24/7/365 operational unit, and it is our only customer. So, the end result is any given developer is on call all the…

So many red flags, time to find a new job. The big one I recognize from a previous job is the 2AM feature implementation. In our shop it generally went like this:

* Sales, having trouble closing a sale, promises a non-existent feature

* Customer tries to use imaginary feature, calls sales to complain

* Developers have to drop what they're doing and implement imaginary feature full steam

edit: formatting

Re: Ask HN: What is your company's on-call rotation like?

#43

Post by snewman »

jeletonskelly wrote: post #14

To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call. Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2…

Minimizing 2am issues, and maintaining an on-call rotation, aren't contradictory. There's no substitute for having someone on call; but you can minimize the number of times they actually get called. This topic is near and dear to my heart.

You talk about testing - that's one side of the coin; the other side is careful alert tuning, (a) to minimize false positives at 2 AM, (b) to catch incipient issues while it's still business hours. (It's useful to think of alerts as just another phase of QA - the one that occurs after you hit production. The sooner you notice a problem, the less damage it causes, to both your customers and your sleep schedule.)

At my workplace, we run a fairly complex system, but we've been able to keep nighttime pager incidents down to I think less than once per quarter, including false alarms. I can't remember the last one. The QA effort isn't overwhelming, either. See http://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/ if you're interested.

Re: Ask HN: What is your company's on-call rotation like?

#44

Post by acveilleux »

fweespeech wrote: post #36

1) We don't keep a formal on call but 3 of us are tied to an automated alert system and whoever has a chance to take care of it, does. We are all full stack devs so generally we can fix it at the time. If its complicated, we can help it hobble and fix it later. 2) We get 3-4 alerts a year that have to be handled before the next business day. 3) As such, there is no real work priority, triage, etc. You resolve it imme…

Has your buffering got any back-pressure at all? Or do you buffer until the disk is full?

Re: Ask HN: What is your company's on-call rotation like?

#45

Post by freehunter »

biot wrote: post #11

For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line shou…

I worked for a small company with a >25 person IT team once, and everyone participated in the on-call. When you were on-call, you were on-call for the entire department. I worked IT security, but I was on-call for networking, servers, code, database, and even power outages. The shifts were for two weeks straight and were non-transferrable, so we couldn't trade off. We needed to be sober, ready to go any time of day or night, and able to be on-site at any of the locations around the city within half an hour. Even if the phone rang at 3am.

I quit after seven months. I wasn't getting paid nearly enough to go through that.

Re: Ask HN: What is your company's on-call rotation like?

#46

Post by freehunter »

darkr wrote: post #34
biot wrote: post #11

For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line shou…

They probably get a lot of repeat work when they tell the patient how much it's costing them.

If you're calling an emergency cardiologist after-hours, you're probably not immediately worried about how much it costs.

Re: Ask HN: What is your company's on-call rotation like?

#48

Post by jeletonskelly »

nrr wrote: post #25
jeletonskelly wrote: post #14

To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call. Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2…

… whereas I'd conversely argue that volumes are also spoken when a company does not require its developers to be on-call. Often, those volumes are written with negative undertones and a narrative that speaks to developers not owning or understanding their software stack. Worse, in an exercise reminiscent of companies offering unlimited PTO, "not on-call," when read between the lines, could really mean "always on-call…

I don't disagree with anything you said here. Organizations should try to hire developers that care about what they build and understand that, as an implied part of the nature of the job, your phone may ring at some unknown time because of an issue. In operations it is certainly implied that you will likely be the first one notified of production issues and you will most likely be the first to know which developer(s) need to be contacted. I am certainly not advocating for a developer to throw their hands up and say "not my problem, I'm not on call" or "it's an ops issue." Those would be very junior or childish reactions and certainly not the qualities of a senior developer.

What my post was getting at more of an observation I have made during interviews where teams had on-call rotations for developers. When I ask "how often does your phone ring during your time on-call" I would get answers that hinted at a deeper issue. Maybe it wasn't always that way, maybe the on-call rotation started with the purest of intentions like you have highlighted, but somewhere along the line management saw that as an opportunity to take shortcuts with testing and quality. So, am I saying that if a company were to make me an excellent offer, but required on-call rotations are they automatically ruled out? No. I am, however, going to be asking some very pointed questions and probing that arrangement quite a bit.

Re: Ask HN: What is your company's on-call rotation like?

#49

Post by hirsin »

herval wrote: post #29
jeletonskelly wrote: post #14

To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call. Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2…

honest question: have you ever managed to work on a company that's either successful or growing very fast and can guarantee "various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2am call"? I can imagine that being possible in consultancies or small scale/load products, but honestly never seen it on anything larger than that - including environments with a mi…

Absolutely - not everything is a service (yet). It's not sexy, but if you only ship your bits every month or every two weeks then you shouldn't have 2am calls. Everything can wait til tomorrow (unless tomorrow you're shipping, in which case I have seen people still awake at 2am, but I consider that a failure).

Re: Ask HN: What is your company's on-call rotation like?

#50

Post by fweespeech »

acveilleux wrote: post #44
fweespeech wrote: post #36

1) We don't keep a formal on call but 3 of us are tied to an automated alert system and whoever has a chance to take care of it, does. We are all full stack devs so generally we can fix it at the time. If its complicated, we can help it hobble and fix it later. 2) We get 3-4 alerts a year that have to be handled before the next business day. 3) As such, there is no real work priority, triage, etc. You resolve it imme…

Has your buffering got any back-pressure at all? Or do you buffer until the disk is full?

It would need to be down for ~96 hours under our heaviest 4 day period in our history for it to fill the disk. All of them would have to also go down simultaneously.

There is backpressure at 85% disk fill (this also is an on-call trigger event since it shouldn't ever happen in practice). Suffice to say, this never happens in the real world without hardware failures.

Disks are cheaper than fatiguing engineers with on-call events and this is accomplished by basically having a nearly empty 3 VM cluster with 512 GB SSD (Raid 1 pair, so 2 disks) each. The load on the rest of the VM host is negligible given its being asynchronously processed from this cluster already, so its just really filling the extra disks on the VM host we dedicated to this purpose.

Just realize we build to 4 9s.

YTD failures 3662/75631129 = 0.00004841921

This doesn't trigger an on-call event because it recovered automatically but the cluster does fail every so often for ~3k events. This is for a single API call.

Post reply on HN