Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Developer on Call

henrikwarne.com

Post reply on HN
Print view View original

31–40 of 246 posts

Re: Developer on Call

#31

Post by hardwaresofton »

beejiu wrote: post #28
hardwaresofton wrote: post #27

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or slightly more depending on what "extra" is), dispense with the usual circus that is performance reviews and start tracking when bad code causes outages. Blame assignment is super counter-productive in the moment of emergency, but it seems like it could be useful…

Blame assignment is counterproductive because it encourages a culture of risk aversion. I suspect highly productive members produce more bugs simply because of the size of their contribution, and that's acceptable in some (perhaps most) businesses.

I think it's fair to say that it's counter-productive for more than one reason.

That said, I think if you want to encourage risk taking, do it directly -- incentivize it with money/prestige or hire more people who take risks (and give them free reign). In recent history more and more "labs"-type positions have been opening up at companies, with the aim being to lure in people who want to do interesting work. As an engineer, a labs position is 1000% more interesting to me than any other senior whatever position because of this ability to take risks and possibly reap large rewards (even if they go to the company).

As far as your point about productive members producing more bugs, you can incentivize/dis-incentivize this by changing how you perform reviews. Incentivize productivity, but not at the cost of introducing more bugs. Shower cash/prestige/autonomy on developers that produce lots of features with low error-counts and people will optimize for that if that's what drives them.

Re: Developer on Call

#32

Post by chrisper »

devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

Why is it called optional if it really isn't?

Re: Developer on Call

#33

Post by hardwaresofton »

badminton1 wrote: post #30
hardwaresofton wrote: post #27

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or slightly more depending on what "extra" is), dispense with the usual circus that is performance reviews and start tracking when bad code causes outages. Blame assignment is super counter-productive in the moment of emergency, but it seems like it could be useful…

An outage could be caused by infrastructure, by library code, by configuration, by increased activity, not necessarily your own software.

Yes, and there's at least two ways you could handle this:

- Have the entire org take a hit

- Penalize the infra team

The thing about blame assignment is that no one wants to get blamed for anything (so like if you try #2 the infra team would likely find someone to blame as quick as possible), which ordinarily makes it pretty toxic but I think you can use it for good here with proper communication and goal-setting (which is of course harder than it sounds).

The case of outages completely caused by infrastructure I think is pretty rare, but if it's really something like S3 going down or whatever just don't count it. If it's that someone pushed an invalid load balancer setting to an ELB, then there's gonna be someone at fault, whether it's the infra team for letting it be possible in the first place or the developer that did it for doing it. Good infra teams try to make bad settings impossible and good settings self-servicable in my experience.

All this said, it really doesn't need to be super heavy handed, I mean don't make some orwellian report-your-neighbor for points system, but enforce accountability and link it to something people care about.

Re: Developer on Call

#34

Post by badminton1 »

devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

> no one has any problem being on call.

How do you know this?

Re: Developer on Call

#35

Post by purple_ducks »

jedberg wrote: post #3

> Pay. People on call should get paid extra for it I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly. > Scheduling. When I have been on call, it has always been one week at a time, I agree wit…

> I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck

So much for 40 hour work week. People would be spinning in their graves if they knew what had become of employer-employee relationship.

Re: Developer on Call

#36

Post by devuo »

chrisper wrote: post #32
devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

Why is it called optional if it really isn't?

it is optional, e.g if you have extra work activities that impede you, family reasons etc. and in these cases it's A-OK! Where it's not so OK it's when there's no reason other than not wanting to do just because you just don't want to be bothered.

This is a problem, because during the interviews the person was repeatedly told we did on call and he/she's ok with it.

For us, as a team, it's important because if you have the right to deploy to live whenever you want (after code review evidently), you have the obligation to keep it. When everyone shares the load, the load is lighter for everyone. And my experience tells me it just makes everyone much much more responsible and professional.

Re: Developer on Call

#37

Post by devuo »

badminton1 wrote: post #34
devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

> no one has any problem being on call. How do you know this?

1:1s, meetings, general team feeling, retrospectives, amount of whining zero to none.

Re: Developer on Call

#38

Post by devuo »

hardwaresofton wrote: post #26
devuo wrote: post #17

At my company, specifically in my team we do on call, and: 1 - it's one week length 2 - it's paid extra 3 - it's optional, but you're a bit of a bad mate if you don't participate 4 - we try to have at least 6 people on rotation to ensure a full month between on call Because we do several changes to production per day, our coverage is around > 99% for all our services and libraries (my team is responsible for about 30…

While what your company is doing is commendable (most don't pay extra or rotate in that fashion) #3 is a red flag for me because it sounds like the overly friendly but in the end passive aggressive and unprofessional atmosphere I've witnessed at startups and midsize companies who pretend they're startups. If on call is optional what's with the social penalty for people not wanting to do it. IMO what companies should…

who said there is a social penalty? There are a number of reasons which I explained in a reply below.

Re: Developer on Call

#39

Post by devuo »

munchbunny wrote: post #29
hardwaresofton wrote: post #26

Earlier quoted context omitted.

While what your company is doing is commendable (most don't pay extra or rotate in that fashion) #3 is a red flag for me because it sounds like the overly friendly but in the end passive aggressive and unprofessional atmosphere I've witnessed at startups and midsize companies who pretend they're startups. If on call is optional what's with the social penalty for people not wanting to do it. IMO what companies should…

I agree on the preference that if it's not really optional, just don't make it optional. On-call also varies a ton between companies. I was technically on-call all of the time in my last job, but it was a low throughput system. I had to be up at odd hours maybe once every 2-3 months. I slept pretty well. If you offered me free meals for the week, I wouldn't mind taking my turn on the watch regularly. This job, I'm on…

evidently the pay must be adjusted to how painful the on call experience is.

as I said it is optional, no harm comes to you for not participating, and we have people with very good reasons for not helping the team support the code they themselves built and deployed themselves to production.

Re: Developer on Call

#40

Post by thedufer »

devuo wrote: post #36
chrisper wrote: post #32

Earlier quoted context omitted.

Why is it called optional if it really isn't?

it is optional, e.g if you have extra work activities that impede you, family reasons etc. and in these cases it's A-OK! Where it's not so OK it's when there's no reason other than not wanting to do just because you just don't want to be bothered. This is a problem, because during the interviews the person was repeatedly told we did on call and he/she's ok with it. For us, as a team, it's important because if you hav…

It sounds like you meant flexible, not optional.
Post reply on HN