Live data from Hacker News

Why should I do production support?

devenbhooshan.wordpress.com

31–40 of 136 posts

Re: Why should I do production support?

#31

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

The cure - Unionisation seriously

heh, I started searching on Spotify...

Re: Why should I do production support?

#32

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

The point about on-call is really critical and really on-point. If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse. It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the nigh…

The companies do this for the money. And the people what work in those companies have no real sense of the risks, or they just care more for the numbers and want to roll the dice.

I would not trust someone who I just woke up at 2am to do something. He/she is mid-sleep. They will be prone to errors, they will be super tired, and I just ruined their next 1.5 days that it will take them to recover from that.

This is not a job where you live boxes where intellect is not needed as much, (strength and stamina will also be affected by a mid-night alarm). You want your folks to be 100% on par, otherwise they may make things worse.

Re: Why should I do production support?

#33
post #27

Earlier quoted context omitted.

Dont most developers do level III support? Ultimately if the first and second lines of defense cannot solve the problem, from everything I've seen, it goes to an Engineer. If it is your product, i'd assume it comes to you. This has been the case at almost every company I've been at.

Why should a company hire two lines of support to shield an engineer from bugs they introduce?

1. Not every support request is the result of a bug.

2. Not every bug is directly related to the product owner/team. Bugs can be introduced from different teams/processes.

3. Requiring senior engineers to do L1 and L2 support is likely a misallocstion of resources when less senior people can handle those issues.

Re: Why should I do production support?

#34
post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any binary. Why are thrall developers less trusted, is it some kind of logic or regulation or just "the way it has always been done" in finance?

Re: Why should I do production support?

#35

If you aren't doing production support, you don't actually know your product. You aren't connected to the pain points your users experience and you miss what is, to your support team and your customers, the glaringly obvious. I would argue all developers should be required to do some support work.

Having skin in the game will make you a better engineer. You’ll get better at the non-sexy things: monitoring, alerting, testing, etc. You better believe a person who is pages for software at 3:00 am has an incentive to make that software more reliable.

But the directors and VP's aren't getting paged at 3am, so the incentives of the organization still go to lower quality software.

Re: Why should I do production support?

#36

If you aren't doing production support, you don't actually know your product. You aren't connected to the pain points your users experience and you miss what is, to your support team and your customers, the glaringly obvious. I would argue all developers should be required to do some support work.

I largely agree. My company has dedicated CS, a second tier 'triage', and well...me. I always prefer if customers just email me directly. CS is a frustrating, cost saving measure. And often I'll get tickets, 3 weeks later, like 'customer said they have an issue.' What's the point again?

Re: Why should I do production support?

#37

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

The point about on-call is really critical and really on-point. If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse. It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the nigh…

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support

And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours.

> Stealing wages from workers by expecting them to be available 24x7 (on-call)

The Tier 3's that are on-call in our environment are on a rotating roster are compensated nicely for being prepared to answer the phone outside business hours. Frequently they don't get called during their week at all and it's free money.

> how is a dev's development ability when they were up at 2:30am on an incident call the night before?

Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff.

There are simple solutions to these problems, and where they matter, they are applied. Granted things are very black and white for us as lives are potentially at stake, but any company that wants to have 24/7 engineers available needs to pay for that kind of support.

Re: Why should I do production support?

#38
post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

Adding to the above 100/% spot on comment; "we" in finance/banking do this because segregation of duties is mandatory, none of that DevOps nonsense ;)

In the off chance that a dev has the unique knowledge to solve a problem, they may get the firefighter/temporary elevated access needed, but will have to document the reason and the dev's actions very very well, because both internal and external auditors will zero in on that.

Re: Why should I do production support?

#39
post #33
post #27

Earlier quoted context omitted.

Why should a company hire two lines of support to shield an engineer from bugs they introduce?

1. Not every support request is the result of a bug. 2. Not every bug is directly related to the product owner/team. Bugs can be introduced from different teams/processes. 3. Requiring senior engineers to do L1 and L2 support is likely a misallocstion of resources when less senior people can handle those issues.

3 is a common mistake, like the one engineers make when they pretend that the hard part of programming is typing

Re: Why should I do production support?

#40

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

Various places I've worked put certain people on-call. They were salaried but somehow paid extra for the time they were on-call. It seemed to be voluntary and the people who did it liked the extra cash.

On the other hand I had a friend at a very large and well known company. He got a job offer and was hired into one department, but he wanted to take a little time off between jobs before he started. They somehow convinced him to start saying there was a holiday coming up and he could take the time off then.

and as soon as he came in he started getting calls 1am 2am 3am etc...

so he left.

And they cajoled him back saying things were different and he finally bought it and went back.

same thing happened again, and he quit for a final time.

One part of the problem was that he was a US citizen working with a bunch of H1B visa folks and the company could get away with that sort of stuff. H1B folks will say yes sir, no sir because their dreams of living in the US are tied to keeping their job at all costs. and then the bad work culture festers.

Post reply on HN