Developer on Call • Hacker News Board

Re: Developer on Call

#91

Post by htanirs » Mon, Dec 03, 2018, 2:43 PM UTC

My take, having managed dev and worked closely with Support, for a growing company.

During Initial days, Initial Dev team is involved in support. And it is amazing. You get real feedback and good insights into how user uses the system. This pays off immensely.

Once the product and team grows, the real need of exclusive support becomes evident. And it becomes quite clear, just like not all support folks can code, not all devs can support. It requires some unique skill set.

Not all requests from customers are critical and even if there are issues, hot fixes need not be necessary. Apart from devs becoming anxious, they may be too eager to comply with requests immediately. Also support requires the team to talk the language user is comfortable with, too much technical communication may not be relevant.

But what has worked for us in growing stage, dev some spending time with support. They can (only) listen to important calls and sharing information between the two teams regularly.

Re: Developer on Call

#92

Post by WrtCdEvrydy » Mon, Dec 03, 2018, 2:45 PM UTC

wink wrote: post #89

jpalomaki wrote: post #71

Kayak.com co-founder Paul English on this topic (2010): "The engineers and I handle customer support. When I tell people that, they look at me like I'm smoking crack. They say, "Why would you pay an engineer $150,000 to answer phones when you could pay someone in Arizona $8 an hour?" If you make the engineers answer e-mails and phone calls from the customers, the second or third time they get the same question, they'…

This sounds so nice in theory. All those pesky developers refusing to fix their horrible code. While in reality at least in my experience the developer would be very much happy to fix the code, it's just that you don't get any time to do that. It's only new features and new products.

Bug fixes don't sell your software unless it's a heavy customer product.

Re: Developer on Call

#93

Post by kylecordes » Mon, Dec 03, 2018, 2:47 PM UTC

time0ut wrote: post #74

As my company has grown (I am an early employee), our on call system has gone through a few changes. A decade ago, we had a physical pager that we handed off every week. The pager was tied to our ticketing system and anyone could create a ticket for it. It worked for the most part, but every now and then the "entire system is down" issue would turn out to by Mary in accounting's internet cable was loose. Then we staf…

I believe there is some value in having a portion of this extend, either in real-time or in terms of attention to hiring experience, to people making key technological/architecture decisions.

I've seen many decisions made where some attention to "what failure modes would such a design have, that might result in human attention at 3 AM?" would lead to different fundamental technology choices. I know that I have made different technology choices and design decisions, based on some early career experience where I was the person who would be paged if the system required human attention.

But if the people making the fundamental technology choices have no experience or exposure to the 3 AM possibility, the trade-off might never be considered until it is too late.

Re: Developer on Call

#94

Post by adetrest » Mon, Dec 03, 2018, 2:48 PM UTC

ownagefool wrote: post #60

patatino wrote: post #46

On my last job somebody told my it's my turn for the support phone. In our case text messages if web based products are down. I never heard of it before, not during my interview or first month, never. I didn't know how to react and said sure, here's my number. After waking up twice during the night I decided to mute my phone completly from 10pm to 7am. Sometimes I woke up and had 50 messages and would try to solve th…

I largely believe developers should be responsible for their work, which includes meeting the support requirements out of hours, and understanding the tooling and instrumentation required to keep their work healthy. However, hiring people and not telling them this is part of the job is insanely unfair. People should get paid for being on-call. Further still, if something is going to wake me up during the night, I wou…

Where it breaks down is when you add a developer are forced to write and deploy shitty code. You know it's broken, told the PO/PM/whoever is in charge, and they refused to listen. Or when new features are more important than fixing bugs.

Re: Developer on Call

#95

Post by roel_v » Mon, Dec 03, 2018, 2:49 PM UTC

sevensor wrote: post #90

I see a fair amount of sentiment in this thread that's averse to what from my perspective is a very light on-call schedule. My former employer was a manufacturing plant that operated 24/7. The engineers (not software, mostly chemical, mechanical, and electrical), all of us carried pagers at all times and were expected to phone in within minutes of being paged, any day or night. A bad enough incident would require you…

"There was no notion of pager duty -- you just always wore a pager."

So how much did that pay?

Re: Developer on Call

#96

Post by mrweasel » Mon, Dec 03, 2018, 2:51 PM UTC

badminton1 wrote: post #22

Waking up a developer should be done as escalation. If resolving an alert requires to "turn it off and on again" you don't need a developer for that. Stress and lack of sleep reduces cognitive performance (what you pay for when hire a developer) and kills employee morale. If you have 2 similar job offers for similar companies, one requires you to be on-call, the other one doesn't... which one would you pick? If you a…

Absolutely. I work for a company that does operations, that include 24/7 on-call, monitoring and incident management.

We basically run software that someone bought from a dev-shop or wrote themselves. 90% of the time restarting a service fixes the issue right now. If it happens more than once, you asses if it's worth waking a developer and what the likely hood of him fixing the bug is. Normally you'd need a new deployment anyway, and you don't really want to do that at 3AM, better to wait until the morning.

You do need to have developer on call, to some extend, but if you have to call them more than once or twice a year, something not right. In those cases, where the same buggy software is a fault for waking you multiple times a week, it not a developer you want on call, it's a project manager or what ever type of middle management is involved.

The issue is that the developers actually do want to fix bug, and write stable software. From a middle management perspective: if someone is up during the night to reboot servers and hand held data imports, then that's a fixes issue, and the developers can focus on new features.

I assure you that if you call up managers at 2AM to tell them that the software they are responsible for has a bug, they will start focusing on stability.

Re: Developer on Call

#97

Post by Cthulhu_ » Mon, Dec 03, 2018, 2:51 PM UTC

wink wrote: post #89

jpalomaki wrote: post #71

Kayak.com co-founder Paul English on this topic (2010): "The engineers and I handle customer support. When I tell people that, they look at me like I'm smoking crack. They say, "Why would you pay an engineer $150,000 to answer phones when you could pay someone in Arizona $8 an hour?" If you make the engineers answer e-mails and phone calls from the customers, the second or third time they get the same question, they'…

This sounds so nice in theory. All those pesky developers refusing to fix their horrible code. While in reality at least in my experience the developer would be very much happy to fix the code, it's just that you don't get any time to do that. It's only new features and new products.

That's exactly a reason why a company should consider having their engineers do support - by the sound of it, there's a support department who - MAYBE - accumulate common questions and send them to a product owner, but in practice they won't and in practice the PO ignores it because there's more important things - in his/her mind - that have precedence.

Re: Developer on Call

#98

Post by toomuchtodo » Mon, Dec 03, 2018, 3:06 PM UTC

krzrak wrote: post #73

patatino wrote: post #46

On my last job somebody told my it's my turn for the support phone. In our case text messages if web based products are down. I never heard of it before, not during my interview or first month, never. I didn't know how to react and said sure, here's my number. After waking up twice during the night I decided to mute my phone completly from 10pm to 7am. Sometimes I woke up and had 50 messages and would try to solve th…

> After waking up twice during the night I decided to mute my phone completly from 10pm to 7am. Have you considered mentioning this to your colleagues, boss or anyone telling you it's your turn? If not - it's really not cool towards your colleagues, boss and organization in general.

It is unfortunate when emotional manipulation is attempted to support wage theft and employee abuse.

Re: Developer on Call

#99

Post by khendron » Mon, Dec 03, 2018, 3:06 PM UTC

I have a love-hate relationship with developer on-call. I see it as necessary and potentially useful, but it often gets abused.

In my last job, it was something you were expected to do, and there was no additional compensation. On the plus side, it did expose you to all parts of the application, areas beyond your usual domain. On the downside, you are not only responsible for your code, but everybody else's as well. It's really shitty to be up at 3 AM fixing a ball that somebody else has dropped.

In addition to the company wide on-call, there was also team on-call, which was a schedule rotation with your team members to be on-call for team-specific issues. The problem was, if you team was small, you ended up being on call a LOT. My team was being continually stripped of members, so for a while I was ending up on-call 24/7 for weeks on end. It was very stressful.

Re: Developer on Call

#100

Post by drb91 » Mon, Dec 03, 2018, 3:06 PM UTC

krzrak wrote: post #73

patatino wrote: post #46

On my last job somebody told my it's my turn for the support phone. In our case text messages if web based products are down. I never heard of it before, not during my interview or first month, never. I didn't know how to react and said sure, here's my number. After waking up twice during the night I decided to mute my phone completly from 10pm to 7am. Sometimes I woke up and had 50 messages and would try to solve th…

> After waking up twice during the night I decided to mute my phone completly from 10pm to 7am. Have you considered mentioning this to your colleagues, boss or anyone telling you it's your turn? If not - it's really not cool towards your colleagues, boss and organization in general.

It’s up to the employer to make expectations clear.