Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Developer on Call

henrikwarne.com

Post reply on HN
Print view View original

11–20 of 246 posts

Re: Developer on Call

#11

Post by js8 »

In some companies, there is a difference between developers (people who create new features) and L2 or L3 support (people who fix bugs and resolve problems for the customer). The trend is not to have this division, and I disagree with that.

I agree that it is good to try both things, and developers should try to be support once in a while, and vice versa. However, I think they are very different mindsets and doing both is making people less productive.

When you're a developer, you need to concentrate on the new feature you're working on only. The better you concentrate, the less bugs and problems you introduce. However, when you fix issues for the customer, it's often punctuated work where you wait for the customer, or investigate the problem, and so you work on many different little things at once. It is not a very good environment to do bigger decisions about architecture changes.

Re: Developer on Call

#12

Post by gaius »

jedberg wrote: post #3

> Pay. People on call should get paid extra for it I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly. > Scheduling. When I have been on call, it has always been one week at a time, I agree wit…

always the on-call engineer, me (the team lead), and then my manager and then their manager.

The best scheme in my experience is to have a 24/7 fully staffed L1 working shifts for initial response triage then waking the appropriate L2 to deal with the actual problem if it isn’t covered in the runbook. No good waking the DBA if you can’t login to the database because a router has crashed and failed to failover. The next morning the L2 guy updates the runbook so if possible if the same thing recurs L1 can just handle it.

What really doesn’t work is having the on-call support also doing out-of-hours work such as maintenance or whatever. That’s a recipe for disaster. People willing to make the sacrifice of on-call are a precious commodity to be used wisely, or they’ll walk.

Paying only in the event of a callout doesn’t work either, that person has still had to decline other activities in order to make themselves available.

Re: Developer on Call

#13

Post by lawnchair_larry »

Neil44 wrote: post #10
jedberg wrote: post #3

> Pay. People on call should get paid extra for it I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly. > Scheduling. When I have been on call, it has always been one week at a time, I agree wit…

If the on call hours were part of your initial contract and renumeration then that's OK. Otherwise they're changing your terms of employment in their favour, taking more of your life than they're paying you for.

Tell me more about this “contract” you speak of. We must work in very different industries if you get anything remotely like what you describe.

Re: Developer on Call

#14

Post by rendall »

js8 wrote: post #11

In some companies, there is a difference between developers (people who create new features) and L2 or L3 support (people who fix bugs and resolve problems for the customer). The trend is not to have this division, and I disagree with that. I agree that it is good to try both things, and developers should try to be support once in a while, and vice versa. However, I think they are very different mindsets and doing bo…

I'm not sure why you're getting downvoted. I think this is a point of view that is at least worth considering.

Personally, I really dislike emergency software debugging. I much prefer the kinds of projects that prioritize testing and code stability so as to obviate the need for an 'on call' developer. In these kinds of projects an emergency call should be so rare as to essentially be never.

Re: Developer on Call

#15

Post by rendall »

What sorts of issues come up in a on-call alarm? When they come up, do you work to mitigate the problem forever, so that this particular alarm never happens again?

I am having a hard time imagining scenarios that need developers to be oncall. Is it a matter of pushing bad code to production?

Re: Developer on Call

#16

Post by wikibob »

jedberg wrote: post #3

> Pay. People on call should get paid extra for it I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly. > Scheduling. When I have been on call, it has always been one week at a time, I agree wit…

Google pays people, both in cash, and/or in compensatory time off. This is specifically called out in the SRE book [0].

They've noted that it's important to pay compensation, both to be fair to the employees, and as a closed-loop feedback mechanism to ensure the business prioritizes fixing pages. This concept of business feedback is also discussed in a chapter of the terrific Seeking SRE book, chapter "Against On-Call: A Polemic" [1].

> Compensation Adequate compensation needs to be considered for out-of-hours support. Different organizations handle on-call compensation in different ways;

> Google offers time-off-in-lieu or straight cash compensation, capped at some proportion of overall salary.

> The compensation cap represents, in practice, a limit on the amount of on-call work that will be taken on by any individual.

> This compensation structure ensures incentivization to be involved in on-call duties as required by the team, but also promotes a balanced on-call work distribution and limits potential drawbacks of excessive on-call work, such as burnout or inadequate time for project work.

[0] https://landing.google.com/sre/sre-book/chapters/being-on-ca... (search Compensation) [1] http://shop.oreilly.com/product/0636920063964.do

Re: Developer on Call

#17

Post by devuo »

At my company, specifically in my team we do on call, and:

1 - it's one week length

2 - it's paid extra

3 - it's optional, but you're a bit of a bad mate if you don't participate

4 - we try to have at least 6 people on rotation to ensure a full month between on call

Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30 of them). We have near zero live incidents, and whenever it does happen the phone rings, it ends up being just some unpredictable spike in load that self heals without intervention.

Because on call is not painful (as it shouldn't be!) and we support each other no one has any problem being on call.

Re: Developer on Call

#18

Post by always_good »

tigershark wrote: post #8

I have done it for 1 year and half and I’ll never, ever do it again. As I learnt, my sleep is worth much more than any amount of money.

I think I have some lasting trauma for the year I was on call.

I still have nightmares that I'm getting woken up into a hellish situation to fix code I've never seen at 3am. Or that I'm out on a date or having a beer or trying to enjoy my life when I get called.

I remember the constant state of anxiety just knowing I could be called. Couldn't even wind down watching a movie much less read a book. I quit when I realized I felt a sense of relief commuting to work the next morning because I wouldn't have to field an emergency by myself.

I also remember fantasizing about being a cafe barista or security guard that year. Waited way too long to get out.

Re: Developer on Call

#20

Post by piquadrat »

> In his book Antifragile, Nassim Nicholas Taleb mentions how Roman engineers had to spend some time under the bridges they built – to ensure they did a good job.

This is a myth, and AFAICT, there is no proof of this being an occurrence in Roman society, at all.

https://www.reddit.com/r/AskHistorians/comments/13t9kn/did_t...

https://skeptics.stackexchange.com/questions/18558/were-roma...

Post reply on HN