Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why should I do production support?

devenbhooshan.wordpress.com

Post reply on HN
Print view View original

81–90 of 136 posts

Re: Why should I do production support?

#81

Post by johnbellone »

quixoticaxolotl wrote: post #8

It sounds very much like the poster is describing an on-call rotation rather than "production support", which is a very different thing altogether. Production support is customer support: responding to chat messages or communications from users. An on-call rotation, on the other hand, involves responding to production incidents and mounting a proper incident response. The Google SRE workbook has a great chapter on th…

I don’t know why you’re being downvoted; these are two entirely different roles.

Re: Why should I do production support?

#83

Post by HenryBemis »

HenryBemis wrote: post #32
dopylitty wrote: post #17

Earlier quoted context omitted.

The point about on-call is really critical and really on-point. If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse. It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the nigh…

The companies do this for the money. And the people what work in those companies have no real sense of the risks, or they just care more for the numbers and want to roll the dice. I would not trust someone who I just woke up at 2am to do something. He/she is mid-sleep. They will be prone to errors, they will be super tired, and I just ruined their next 1.5 days that it will take them to recover from that. This is not…

Type: live = "lift boxes" (just spotted that, apologies I can't edit and correct)

Re: Why should I do production support?

#84

Post by machello13 »

nineteen999 wrote: post #37
dopylitty wrote: post #17

Earlier quoted context omitted.

The point about on-call is really critical and really on-point. If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse. It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the nigh…

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours. > S…

> Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff.

Every company should do this but none I've worked at do. To be honest, I just take the makeup time myself.

Re: Why should I do production support?

#85

Post by jp0d »

It depends on what and how much one is learning from it. I once interviewed a guy for an ETL developer role. He was supporting an ETL application. In four years, all he had done was restart the application in case of any issues and it somehow worked for him. I've seen my fair share of such cases.

Re: Why should I do production support?

#86

Post by tjpnz »

nimrody wrote: post #72
afberg wrote: post #60

Earlier quoted context omitted.

I think there's only one way to solve this — which I've been unsuccessfully advocating for at my current company — and that is paid voluntary on call schedules. It creates an actual market for on call work where engineers can simply say no to the extra cash if they don't like work taking up their nights and weekends. If the company is having trouble with no engineers wanting to be on call the pay is simply too low an…

But does it give engineers a good incentive to improve product quality and reduce the number of production incidents? Where I work we are not on-call. Nevertheless, I try to help the ops team when they encounter issues. This does make you improve logging and error handling since you know it takes a lot more time when it's difficult to filter logs for the interesting events. Engineers not exposed to production issues…

Our on-call system gives everyone an allowance per service plus additional time off if you're on-call on weekends and public holidays. You get both regardless of whether you get paged. Getting paid to do nothing is a great incentive for pushing quality, especially when you're on-call for more than one service in a given week.

Re: Why should I do production support?

#87

Post by throwaway0a5e »

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

I'm not gonna advocate for having engineers directly field customer calls at 3am (because that's a recipe for unhappy customers) but as someone who fields customer support calls/tickets as a large part of my job I feel very comfortable saying that a "sizeable enough to be a big problem" will not build systems that are supportable without engineering escalations unless they get support trying to escalate an issue or conference them in at 3am from time to time.

Re: Why should I do production support?

#89

Post by tasogare »

hinkley wrote: post #49
crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

> "do you even realise what a service to humanity you are doing!". I don’t know who needs to hear this besides me twenty years ago, but if you want to do charity then go home at 6pm and volunteer at a real charity . Don’t do it for a wannabe robber baron who will not share with you. Don’t do it for someone where even an emotional payoff is years away or may never come. Find something else you care about and help some…

Well even "real charity", including very famous ones, are quite shady too: volunteers do work for free while donations goes towards C-exec 5-6 digits salaries.

Re: Why should I do production support?

#90

Post by 0xbadcafebee »

sdevonoes wrote: post #65

I like to write software but I don't want to be on-call if the software I wrote breaks at 3am in the morning. I do take my job with professionalism, I do write tests for most of it (not 100% coverage, but 100% coverage of the critical parts), I do monitoring (and answer and fix alerts if they happen during working hours) and I don't deploy on Fridays (and don't allow people to deploy on Fridays). My code will crash s…

This attitude almost always turns into the following:

On-call: "Hey devs, I'm being woken up at 3AM because your app sucks. Please fix it." Devs: "Sure, no problem."

4 months go by

On-call: "These alerts are still coming in at 3AM. Did you fix the issue?" Dev: "We have a lot of work, we can't dedicate all our time to some minor problems, we have a deadline."

Next week, Devs are put on-call.

The alerts are fixed in two weeks. Site reliability goes up. Apps suddenly become more resilient to failure.

Honestly, the whole attitude of not wanting to work more than 8 hours is privilege. Most of the rest of the world works long hours. As a dev, you get a good salary and a job you don't have to break your body to do. The least you can do is be completely responsible for your own code.

And it helps you as an engineer. Like the article points out, it creates empathy for the users and product support engineers, it helps you improve architecture and app design, and it helps you understand different failure domains. You won't learn all that on your own time, especially without the scale of production.

Post reply on HN