Developer on Call
henrikwarne.com
Developer on Call
1–10 of 246 posts
Re: Developer on Call
#2> These days we are almost never woken up by phone calls, because we get the alarms from the monitoring before most customers notice a problem.
Interesting. If I get woken up while on call, it's never by another person - rather it's by an alarm. Should I start deferring these alarms until next morning to get more sleep?
For context, I am new to being on call - this process was in place before I started.
Re: Developer on Call
#3> Pay. People on call should get paid extra for it
I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly.
> Scheduling. When I have been on call, it has always been one week at a time,
I agree with this one. At Netflix we tried a bunch of different schemes, going from just a few hours on call at a time to a week at a time. The week seemed to work out best for everyone.
> Escalation. There should always be an escalation path if there is a real crisis.
At Netflix, our escalation path was always the on-call engineer, me (the team lead), and then my manager and then their manager. It almost never got past the first engineer, and in the rare cases it did, pretty much everyone on the team was ok with getting a call at any time to help in a crisis, so usually we'd just call whoever would have the most relevant expertise for the current issue. Oftentimes another one of us was already on the call listening anyway. It rarely rolled up to me.
My point being, beyond the rigidity of one person being designated on call, if you really want it to work well you need to be flexible and trust that your team is made of competent people that you can rely on, and they need to be cool with getting a call when they aren't on call, assuming that they might call on you one day.
Re: Developer on Call
#4> These days we are almost never woken up by phone calls, because we get the alarms from the monitoring before most customers notice a problem. Interesting. If I get woken up while on call, it's never by another person - rather it's by an alarm. Should I start deferring these alarms until next morning to get more sleep? For context, I am new to being on call - this process was in place before I started.
This reads to me as "I'm never woken up by phone calls from customers (or managers angry that customers are complaining about it being broken), because our alerts will wake me up first".
If the phone rings, you answer it. If, after some assessment, it can wait until morning, you leave it til morning. Then you make sure it won't wake the next person up while they are on call.
Re: Developer on Call
#5> These days we are almost never woken up by phone calls, because we get the alarms from the monitoring before most customers notice a problem. Interesting. If I get woken up while on call, it's never by another person - rather it's by an alarm. Should I start deferring these alarms until next morning to get more sleep? For context, I am new to being on call - this process was in place before I started.
No you should not. :) If you slept through alarms on purpose I would be livid with you. Unless they are poor alarms that don't indicate actual issues (in which case you should turn off those alarms) I would assume that an alarm is a leading indicator and take care of it. Ideally you never want the customer to know you had an issue.
Re: Developer on Call
#6My current teams on call is pretty taxing. Two weeks on, two weeks off (only myself and my tech lead in the team at the moment), but our alerting is pretty good.
The places it falls down are where we interface with other teams who aren't on call for their systems and for them a weekend long outage is "acceptable".
Re: Developer on Call
#7>Pay. People on call should get paid extra for it. There is a significant impact on your life if you have to be ready to log on and trouble shoot issues at any time during a week, so you deserve to be compensated for that. I think the best system is when you both get a fixed amount just for being on call, whether there are incidents or not, and you also get paid every time you get called out. Getting time off in lieu is also a possibility.
Depending on where you live this may not be realistic. In Japan for example it's quite common for companies to put their engineers on call without compensation - even if it's a legal gray area. I was once on a team that had to threaten management with a lawyer when they tried to propose this, but I have a feeling the majority of workers here would just swallow it.
There are other logistical factors that need to be considered which this article makes no mention of. What happens when someone who is on-call lives/commutes through an area with patchy cellphone coverage? What do you do regarding alcohol consumption?
Re: Developer on Call
#8I have done it for 1 year and half and I’ll never, ever do it again.
As I learnt, my sleep is worth much more than any amount of money.
Re: Developer on Call
#9I have done it for 1 year and half and I’ll never, ever do it again. As I learnt, my sleep is worth much more than any amount of money.
One solution to this problem is to "follow-the-sun" if you have multiple teams across the globe.
Re: Developer on Call
#10> Pay. People on call should get paid extra for it I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly. > Scheduling. When I have been on call, it has always been one week at a time, I agree wit…
If the on call hours were part of your initial contract and renumeration then that's OK. Otherwise they're changing your terms of employment in their favour, taking more of your life than they're paying you for.