Developer on Call • Hacker News Board

Re: Developer on Call

#41

Post by js8 » Mon, Dec 03, 2018, 10:15 AM UTC

varjag wrote: post #24

js8 wrote: post #11

In some companies, there is a difference between developers (people who create new features) and L2 or L3 support (people who fix bugs and resolve problems for the customer). The trend is not to have this division, and I disagree with that. I agree that it is good to try both things, and developers should try to be support once in a while, and vice versa. However, I think they are very different mindsets and doing bo…

Not sure about this. Separating dev work like that is a risk to get architecture astronauts in low stress position with a bunch of peons suffering for their sins in the trenches. Concentration is not a good argument, bug fixing requires no less of that and is often compounded by time pressure.

I agree and I think couple months rotation is ideal, actually. I certainly do not advocate architects who do not code etc.

Of course, it depends on person. Some people are really good architects/developers and you don't want them to spend their days talking customers through issues. On the other hand, some people are more comfortable in support and that's good too.

And I think if a bug fix requires larger rearchitecture of the thing (that is, the root of the problem is more conceptual than just incorrect code), then it's better addressed by temporary patch and doing it properly in development cycle.

Re: Developer on Call

#42

Post by eertami » Mon, Dec 03, 2018, 10:15 AM UTC

purple_ducks wrote: post #35

jedberg wrote: post #3

> Pay. People on call should get paid extra for it I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly. > Scheduling. When I have been on call, it has always been one week at a time, I agree wit…

> I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck So much for 40 hour work week. People would be spinning in their graves if they knew what had become of employer-employee relationship.

Even 40 sounds too high to be honest, I've only been in the industry 4 years but already refuse anything more than 30hrs.

Re: Developer on Call

#43

Post by EZ-E » Mon, Dec 03, 2018, 10:18 AM UTC

I was that (single) dev on call for an entier app backend, and I'm not sure I would do it again. You might always get that call 24/7 will give you PTSD of whatever phone ringtone you setup, it's an easy anxiety trigger. Especially if you interface with unstable third parties which will make some calls unavoidable (cough Firebase DB cough )

Re: Developer on Call

#44

Post by pjc50 » Mon, Dec 03, 2018, 10:21 AM UTC

ascar wrote: post #23

rendall wrote: post #14

Earlier quoted context omitted.

I'm not sure why you're getting downvoted. I think this is a point of view that is at least worth considering. Personally, I really dislike emergency software debugging. I much prefer the kinds of projects that prioritize testing and code stability so as to obviate the need for an 'on call' developer. In these kinds of projects an emergency call should be so rare as to essentially be never .

That kind of sounds like fantasy for complex high throughput systems. As it was also mentioned in the article, the usual on-call response is a rollback to the last working version of the correct deployable. This is pretty easy to identify for someone in the team developing the service.

There's a large category of complex systems where you have to get it right in advance because of the consequences of problems; anything avionics or real-time, for example. Or you can lose a lot of money without blowing things up but still faster than the on-call humans can respond: https://dougseven.com/2014/04/17/knightmare-a-devops-caution...

(High-reliability engineering is very much a different, more expensive, less agile culture from software startups, and I worry that the culture is bleeding across in inappropriate ways. The "self-driving" car with a "safety driver" is an extreme example of this: an on-call human that's supposed to respond to operational problems in an extremely short timeframe, but also provides an opportunity to blame the human rather than the software)

Re: Developer on Call

#45

Post by jnsaff2 » Mon, Dec 03, 2018, 10:23 AM UTC

devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

I think you should try to tune out the self healing spikes from your alerts as Alarm Fatigue is real and your mind gets programmed to treat it as yet another spike and either not taking it seriously or assuming that 'adding more resources' is going to be the solution instead of properly diagnosing the problem.

Re: Developer on Call

#46

Post by patatino » Mon, Dec 03, 2018, 10:25 AM UTC

On my last job somebody told my it's my turn for the support phone. In our case text messages if web based products are down. I never heard of it before, not during my interview or first month, never.

I didn't know how to react and said sure, here's my number. After waking up twice during the night I decided to mute my phone completly from 10pm to 7am.

Sometimes I woke up and had 50 messages and would try to solve the problem from home before heading in the office. Sometimes a coworker was already on it if they were in the office early.

One time my boss asked in the morning if I didn't get any messages. I said sure I did, but I was sleeping, I don't get paid to get up in the middle of the night. He didn't say anything and I would keep turning my notifications off :)

Re: Developer on Call

#47

Post by pjc50 » Mon, Dec 03, 2018, 10:27 AM UTC

hardwaresofton wrote: post #27

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or slightly more depending on what "extra" is), dispense with the usual circus that is performance reviews and start tracking when bad code causes outages. Blame assignment is super counter-productive in the moment of emergency, but it seems like it could be useful…

Remember that you can either have an inquiry that finds out what happened or one that assigns blame, but not both; if there is a hint that real blame with consequences will be assigned people will clam up, hide the evidence, or even start destroying evidence and framing their colleagues.

About the most consequential thing you can get away with without wrecking trust is mild humourous social shame like making someone wear a silly hat. And for things which have gone badly wrong that seems inappropriate.

This kind of thing is what blogger Alex Harrowell brilliantly coined "Coasian hell" megaprojects don't work: http://www.harrowell.org.uk/blog/2018/01/31/in-the-eternal-i...

Re: Developer on Call

#48

Post by pjc50 » Mon, Dec 03, 2018, 10:29 AM UTC

lawnchair_larry wrote: post #13

Neil44 wrote: post #10

Earlier quoted context omitted.

If the on call hours were part of your initial contract and renumeration then that's OK. Otherwise they're changing your terms of employment in their favour, taking more of your life than they're paying you for.

Tell me more about this “contract” you speak of. We must work in very different industries if you get anything remotely like what you describe.

If you're not getting a written contract you're not working in the industry, you're the victim of a fraud.

Re: Developer on Call

#49

Post by EZ-E » Mon, Dec 03, 2018, 10:29 AM UTC

always_good wrote: post #18

tigershark wrote: post #8

I have done it for 1 year and half and I’ll never, ever do it again. As I learnt, my sleep is worth much more than any amount of money.

I think I have some lasting trauma for the year I was on call. I still have nightmares that I'm getting woken up into a hellish situation to fix code I've never seen at 3am. Or that I'm out on a date or having a beer or trying to enjoy my life when I get called. I remember the constant state of anxiety just knowing I could be called. Couldn't even wind down watching a movie much less read a book. I quit when I realiz…

> I think I have some lasting trauma for the year I was on call.

I can relate, I get big anxiety rush anytime my phone rings ever since.

Re: Developer on Call

#50

Post by pommers » Mon, Dec 03, 2018, 10:46 AM UTC

wikibob wrote: post #21

pommers wrote: post #6

My current teams on call is pretty taxing. Two weeks on, two weeks off (only myself and my tech lead in the team at the moment), but our alerting is pretty good. The places it falls down are where we interface with other teams who aren't on call for their systems and for them a weekend long outage is "acceptable".

This is not sustainable, you will burn out in the long run and could take an extended period of time to recover. You are risking your health. I suggest you look at the on-call chapters in the SRE book, SRE Workbook, and Seeking SRE. The solution is primarily to include the development team in the on-call rotation (you build it - you run it). This can be very hard to do politically.

Fully aware that it is unsustainable. Being a small team, we chose to maximize the time between on call stints. We will revisit the decision early next year.

We're hiring people with on call being something that is part of the position they are taking.

As for the other teams, we're working on the politics to get them to support there systems, and looking at alternatives to using them if they don't.