Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Developer on Call

henrikwarne.com

Post reply on HN
Print view View original

21–30 of 246 posts

Re: Developer on Call

#21

Post by wikibob »

pommers wrote: post #6

My current teams on call is pretty taxing. Two weeks on, two weeks off (only myself and my tech lead in the team at the moment), but our alerting is pretty good. The places it falls down are where we interface with other teams who aren't on call for their systems and for them a weekend long outage is "acceptable".

This is not sustainable, you will burn out in the long run and could take an extended period of time to recover. You are risking your health.

I suggest you look at the on-call chapters in the SRE book, SRE Workbook, and Seeking SRE.

The solution is primarily to include the development team in the on-call rotation (you build it - you run it). This can be very hard to do politically.

Re: Developer on Call

#22

Post by badminton1 »

Waking up a developer should be done as escalation.

If resolving an alert requires to "turn it off and on again" you don't need a developer for that.

Stress and lack of sleep reduces cognitive performance (what you pay for when hire a developer) and kills employee morale.

If you have 2 similar job offers for similar companies, one requires you to be on-call, the other one doesn't... which one would you pick?

If you are having a very bad on-call week and a recruiter reaches to you, you will be more likely to talk to them, or will be more likely to ask for a raise or just quit.

The "skin in the game" argument sucks. Developers are not solely responsible for software quality. Deadlines are often not set by developers.

Re: Developer on Call

#23

Post by ascar »

rendall wrote: post #14
js8 wrote: post #11

In some companies, there is a difference between developers (people who create new features) and L2 or L3 support (people who fix bugs and resolve problems for the customer). The trend is not to have this division, and I disagree with that. I agree that it is good to try both things, and developers should try to be support once in a while, and vice versa. However, I think they are very different mindsets and doing bo…

I'm not sure why you're getting downvoted. I think this is a point of view that is at least worth considering. Personally, I really dislike emergency software debugging. I much prefer the kinds of projects that prioritize testing and code stability so as to obviate the need for an 'on call' developer. In these kinds of projects an emergency call should be so rare as to essentially be never .

That kind of sounds like fantasy for complex high throughput systems. As it was also mentioned in the article, the usual on-call response is a rollback to the last working version of the correct deployable. This is pretty easy to identify for someone in the team developing the service.

Re: Developer on Call

#24

Post by varjag »

js8 wrote: post #11

In some companies, there is a difference between developers (people who create new features) and L2 or L3 support (people who fix bugs and resolve problems for the customer). The trend is not to have this division, and I disagree with that. I agree that it is good to try both things, and developers should try to be support once in a while, and vice versa. However, I think they are very different mindsets and doing bo…

Not sure about this. Separating dev work like that is a risk to get architecture astronauts in low stress position with a bunch of peons suffering for their sins in the trenches. Concentration is not a good argument, bug fixing requires no less of that and is often compounded by time pressure.

Re: Developer on Call

#25

Post by ascar »

js8 wrote: post #11

In some companies, there is a difference between developers (people who create new features) and L2 or L3 support (people who fix bugs and resolve problems for the customer). The trend is not to have this division, and I disagree with that. I agree that it is good to try both things, and developers should try to be support once in a while, and vice versa. However, I think they are very different mindsets and doing bo…

Adding an extra layer of people between the customer and developer is certainly a good idea as it requires a completely different skill set to communicate with a customer and troubleshoot their issue than for being a developer. Like I always ask my mom before I even begin to troubleshoot her problems "Have you tried turning it on and off again?" "Did you really turn it on and off again and not only open/closed the display?" Our support people, when asked for a forward to a developer afaik often jokingly say "you don't want to talk with one of our developers" and I think it's true for most issues.

Splitting the people that write software, fix bugs for that software and make sure their system is available throughout the night is terrible from a responsibility point of view. It makes you not give a crap, while developing new features. That's just human nature.

Re: Developer on Call

#26

Post by hardwaresofton »

devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

While what your company is doing is commendable (most don't pay extra or rotate in that fashion) #3 is a red flag for me because it sounds like the overly friendly but in the end passive aggressive and unprofessional atmosphere I've witnessed at startups and midsize companies who pretend they're startups. If on call is optional what's with the social penalty for people not wanting to do it.

IMO what companies should be doing is paying extra per hour until they get people that want to do it. As in, increase the price they pay "extra" until someone decides to give up their free time outside of normal development hours.

Re: Developer on Call

#27

Post by hardwaresofton »

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or slightly more depending on what "extra" is), dispense with the usual circus that is performance reviews and start tracking when bad code causes outages.

Blame assignment is super counter-productive in the moment of emergency, but it seems like it could be useful for measuring developer effectiveness, incentivizing shipping features but also shipping features with a small amount of bugs. I have written my fair share of bugs that have snuck into test/staging/production (I'm a prolific at writing bugs), but that's the kind of thing that should come up in a yearly review (and I expect it to) and hurt my chances for raises/promotions, instead of the bullshit musical chairs, politics and level/rank setting (how many more years until you reach Senior Staff Software Engineer IV with distinction again?) that happens right now.

Also my ideal on-call situation (which probably doesn't exist):

- optional

- paid by supply/demand (price per hour on-call increases until someone decides to do it)

Companies should go back to hiring competent night staff for truly critical business processes and paying them whatever is appropriate. The on-call system as it sits now is heavily tilted in favor of business at the expense of employees -- the attention of a $100/hr+ professional for free, or some small percentage of the actual cost.

Also BTW if you write software and don't care that it's bug-free or don't take responsibility for it, you're a bad software engineer/developer. You don't have to be passionate about code but being a professional generally means producing quality work, and quality work is reasonably robust whenever it can be. One of the differences between a junior and senior software engineer is knowledge of what constitutes enough "quality" in context.

Re: Developer on Call

#28

Post by beejiu »

hardwaresofton wrote: post #27

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or slightly more depending on what "extra" is), dispense with the usual circus that is performance reviews and start tracking when bad code causes outages. Blame assignment is super counter-productive in the moment of emergency, but it seems like it could be useful…

Blame assignment is counterproductive because it encourages a culture of risk aversion. I suspect highly productive members produce more bugs simply because of the size of their contribution, and that's acceptable in some (perhaps most) businesses.

Re: Developer on Call

#29

Post by munchbunny »

hardwaresofton wrote: post #26
devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

While what your company is doing is commendable (most don't pay extra or rotate in that fashion) #3 is a red flag for me because it sounds like the overly friendly but in the end passive aggressive and unprofessional atmosphere I've witnessed at startups and midsize companies who pretend they're startups. If on call is optional what's with the social penalty for people not wanting to do it. IMO what companies should…

I agree on the preference that if it's not really optional, just don't make it optional.

On-call also varies a ton between companies. I was technically on-call all of the time in my last job, but it was a low throughput system. I had to be up at odd hours maybe once every 2-3 months. I slept pretty well. If you offered me free meals for the week, I wouldn't mind taking my turn on the watch regularly.

This job, I'm on call maybe one week every two months on a high throughput system, and even though it's only half of the day (we have an overseas team to take the night shift), it's generally acknowledged in the team that your sleep takes a hit and you get no real work done that week. If this were an optional part of my job, you'd have to pay me double for the week (basically a 10% raise).

Re: Developer on Call

#30

Post by badminton1 »

hardwaresofton wrote: post #27

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or slightly more depending on what "extra" is), dispense with the usual circus that is performance reviews and start tracking when bad code causes outages. Blame assignment is super counter-productive in the moment of emergency, but it seems like it could be useful…

An outage could be caused by infrastructure, by library code, by configuration, by increased activity, not necessarily your own software.
Post reply on HN