Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Developer on Call

henrikwarne.com

Post reply on HN
Print view View original

61–70 of 246 posts

Re: Developer on Call

#61

Post by devuo »

jnsaff2 wrote: post #45
devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

I think you should try to tune out the self healing spikes from your alerts as Alarm Fatigue is real and your mind gets programmed to treat it as yet another spike and either not taking it seriously or assuming that 'adding more resources' is going to be the solution instead of properly diagnosing the problem.

Beeping is a rare event in my team, might happen 1-3 times in a week, or there are weeks it doesn't beep at all, and we work to keep it that way.

Re: Developer on Call

#62

Post by zimpenfish »

devuo wrote: post #37
badminton1 wrote: post #34

Earlier quoted context omitted.

> no one has any problem being on call. How do you know this?

1:1s, meetings, general team feeling, retrospectives, amount of whining zero to none.

But do you keep track of how many people might not join because of the on-call? Or have exit interviews that check whether being on-call was a contributing factor?

Re: Developer on Call

#63

Post by devuo »

hardwaresofton wrote: post #55
devuo wrote: post #38

Earlier quoted context omitted.

who said there is a social penalty? There are a number of reasons which I explained in a reply below.

> 3 - it's optional, but you're a bit of a bad mate if you don't participate This seems to clearly state that you have a negative opinion of you for turning down extra work that is clearly undesirable. I assume that you're not the only one on your team that feels this way either, that's what I meant by social penalty. Depending on who you ask it may not be a penalty but that would mean everyone on the team has to thi…

team effort is a thing, and we want everyone to pull their own weight. In my team we work at most 40h week and on call alerts are rare events, we work to keep it that way and we need solid team spirit to do that.

There are other companies to work at, and we make our expectations clear before the person is hired in relation to on call.

Re: Developer on Call

#64

Post by devuo »

zimpenfish wrote: post #62
devuo wrote: post #37

Earlier quoted context omitted.

1:1s, meetings, general team feeling, retrospectives, amount of whining zero to none.

But do you keep track of how many people might not join because of the on-call? Or have exit interviews that check whether being on-call was a contributing factor?

If people don't join because of that, then I'd say that's a filter and I'm ok with that. In the Lisbon office of the company I work, on call was not a contributing factor for the people who have left. The vast majority was because they wanted to work in a different country, and not a problem with the company per se.

Re: Developer on Call

#65

Post by isostatic »

I don't mind being called up, as long as I 1) can choose not to answer the call 2) Am empowered to fix the problem

A common system at our work is that support calls are first passed to a 24 hour helpdesk, who have a decent clue, and have access to fault finding documentation that the development team writes. If X do Y etc.

Only if that documentation fails does it get escalated to the developers. This encourages the developers to write good documentation, and ensures that trivial fixes can be sorted without calling out the developers.

Personally I love it when I get called for 5 minutes on a Saturday morning, tell them to turn it off and on again, and claim a half day off in compensation.

Re: Developer on Call

#66

Post by sheepmullet »

hardwaresofton wrote: post #26
devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

While what your company is doing is commendable (most don't pay extra or rotate in that fashion) #3 is a red flag for me because it sounds like the overly friendly but in the end passive aggressive and unprofessional atmosphere I've witnessed at startups and midsize companies who pretend they're startups. If on call is optional what's with the social penalty for people not wanting to do it. IMO what companies should…

> If on call is optional what's with the social penalty for people not wanting to do it.

Because many optional activities have an impact on your peers and they are unlikely to judge you strictly based upon your job duties?

Re: Developer on Call

#67

Post by sheepmullet »

piquadrat wrote: post #20

> In his book Antifragile, Nassim Nicholas Taleb mentions how Roman engineers had to spend some time under the bridges they built – to ensure they did a good job. This is a myth, and AFAICT, there is no proof of this being an occurrence in Roman society, at all. https://www.reddit.com/r/AskHistorians/comments/13t9kn/did_t... https://skeptics.stackexchange.com/questions/18558/were-roma...

I’m not sure your two links have any substance to them.

A few history buffs couldn’t find anything to support it....

I’d be interested to know how they did test bridges etc.

Re: Developer on Call

#68

Post by ascar »

patatino wrote: post #46

On my last job somebody told my it's my turn for the support phone. In our case text messages if web based products are down. I never heard of it before, not during my interview or first month, never. I didn't know how to react and said sure, here's my number. After waking up twice during the night I decided to mute my phone completly from 10pm to 7am. Sometimes I woke up and had 50 messages and would try to solve th…

My contract with a company that had on call read somewhere along the lines of 9 to 5 are the core working times and all work necessary outside these times, including Saturday and Sunday, is already remunerated in the base salary.

In this case I was being paid for it and I only heard about it in the first week after starting. I asked "how often do I have to work on weekends", but not "will i be on call" in the contract negotiations. Definitely a question I will ask in the future. Although I think being on call is a good thing.

Re: Developer on Call

#69

Post by ascar »

pjc50 wrote: post #44
ascar wrote: post #23

Earlier quoted context omitted.

That kind of sounds like fantasy for complex high throughput systems. As it was also mentioned in the article, the usual on-call response is a rollback to the last working version of the correct deployable. This is pretty easy to identify for someone in the team developing the service.

There's a large category of complex systems where you have to get it right in advance because of the consequences of problems; anything avionics or real-time, for example. Or you can lose a lot of money without blowing things up but still faster than the on-call humans can respond: https://dougseven.com/2014/04/17/knightmare-a-devops-caution... (High-reliability engineering is very much a different, more expensive, l…

High realiabity and high availability are not the same thing though. There are still problems in aviation, like the dreamliner who had to be restarted every x days or it would go full system shutdown. In these kind of systems you often sacrifice availability for reliability. You also sacrifice progress for reliability, which is absolutely the right thing to do for these projects.

An on call engineer shouldn't be the solution for bad reliability, because as you said it doesn't help. He is primarily there for availability.

Instead of high throughput I maybe should've said highly available systems.

Re: Developer on Call

#70

Post by borghildhedda »

1. for every week of oncall a developer should get 1 day off.

2. If developer had to work nights, he should be compensation with additional days off.

3. no payment would reduce the stress so we should not ask for payment compensation.

4. We as developers have let this on us too easily, to eliminate stress devs must form a group and do not sign contracts which do not provide automatic day off for oncall.

Post reply on HN