Why should I do production support? • Hacker News Board

Re: Why should I do production support?

#71

Post by magicalhippo » Tue, Sep 22, 2020, 11:25 AM UTC

zeckalpha wrote: post #61

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

If the engineers aren’t oncall, who is? Is it okay to exploit non-engineers? If anything, it is less exploitative to have those who are empowered to improve their situation oncall.

Our support folks handle the on-call support. They do one week each on rotation, they get some extra compensation and the following friday off.

If it's a serious issue they can't handle they might wake up one of us programmers, but usually they can find some temporary fix or workaround until the next morning.

Re: Why should I do production support?

#72

Post by nimrody » Tue, Sep 22, 2020, 11:25 AM UTC

afberg wrote: post #60

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

I think there's only one way to solve this — which I've been unsuccessfully advocating for at my current company — and that is paid voluntary on call schedules. It creates an actual market for on call work where engineers can simply say no to the extra cash if they don't like work taking up their nights and weekends. If the company is having trouble with no engineers wanting to be on call the pay is simply too low an…

But does it give engineers a good incentive to improve product quality and reduce the number of production incidents?

Where I work we are not on-call. Nevertheless, I try to help the ops team when they encounter issues. This does make you improve logging and error handling since you know it takes a lot more time when it's difficult to filter logs for the interesting events.

Engineers not exposed to production issues and customers will never understand why you need these extra measures.

Re: Why should I do production support?

#73

Post by brailsafe » Tue, Sep 22, 2020, 11:33 AM UTC

I feel like the author is referring more specifically to troubleshooting issues when production goes down, which I'm fine with if I have no other people asking me for updates. But, I burnt out at my lost job trying to do support, because there was individual support baked into every contract for our SDK and not remotely enough people to handle it all. I was hired as a software developer, not a customer support person, and they are not the same thing. It's unfortunate, because it was my first and highest paying gig after realizing that I also have ADHD, and it was a good company. Thing is, if I have a problem to solve or task to complete, I'm not going to think about how long it's been since I replied to whoever about their pet issue. I'm just going to zone in on my thing, and if the guy next to me doesn't break me out of it by cracking eggs on the desk and burping, then I'll stay on that thread till it's done. That's how my brain works, and expecting otherwise is naive. Anyway, this constant context switching and battling my apparent insufficiency killed my spirit for the work and I turned into a blob of productivity. It's as stupid as expecting me to cook while programming, because either the food, myself, or the code will get burnt.

Re: Why should I do production support?

#74

Post by mtberatwork » Tue, Sep 22, 2020, 11:40 AM UTC

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

> There's no excuse for burning out engineers (esp. unsuspecting youngsters) by pushing them into ungodly hours of work ruining their health among other things while trying to constantly tell them - "do you even realise what a service to humanity you are doing!".

In the US, this is happening across the board and not just in tech. The expectation to always be available [often without monetary compensation] is sadly the new normal. Without strong labor laws in place, this implicit form of exploitation will never cease.

Re: Why should I do production support?

#75

Post by nineteen999 » Tue, Sep 22, 2020, 11:40 AM UTC

piokoch wrote: post #68

nineteen999 wrote: post #37

Earlier quoted context omitted.

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours. > S…

"Frequently they don't get called during their week at all and it's free money." That's, I think, is a wrong perspective, when people are on call, they have to be somewhere near their computer/internet connection and be ready to work (so it is not just you can go to a party and if call happens do some quick fix in a toilet). On-calls cannot do what they want with their time, so they don't get money for free.

Yeah this is true although our compensation for their time is pretty generous. As a team we are pretty flexible too and if somebody has an event they want to go to they can always find someone else on the team to cover or swap with them for the night.

Re: Why should I do production support?

#76

Post by mtberatwork » Tue, Sep 22, 2020, 11:42 AM UTC

MattGaiser wrote: post #19

dopylitty wrote: post #17

Earlier quoted context omitted.

The point about on-call is really critical and really on-point. If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse. It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the nigh…

They should also allow time and budget for building an application that can run 24/7 without too many errors. "We need [insert thing manager asks for here] immediately" has consequences.

It doesn't have consequences though, which is why it keeps happening.

Re: Why should I do production support?

#77

Post by mtberatwork » Tue, Sep 22, 2020, 11:47 AM UTC

jake_morrison wrote: post #69

A lot of the pain around production support is easily solved by having staff in multiple time zones. A good structure is to have first line support be relatively generic ops people. They can handle problems related to infrastructure, e.g. hardware failures, network problems, or issues that can be handled by adding resources. The deployment process should be consistent enough across applications that they can e.g. rol…

Convincing folks at the top of food chain that more staff is needed is one of the most difficult things to do.

Re: Why should I do production support?

#78

Post by Galanwe » Tue, Sep 22, 2020, 11:50 AM UTC

Ma8ee wrote: post #43

Galanwe wrote: post #30

Earlier quoted context omitted.

The cure - proper labor law. Seriously

Which is one of the things that Unions are good at pushing for. Who do you think create laws? A single guy walking up to his Member of Congress.

I guess it depends on everyone's personal experience with unions.

From where I come from, unions are mainly a way for lazy employees to get immunity while doing nothing all day long.

I would still maintain that if you live in a country where it's legal to be called at any time, any day, then you have a third world class labor law - go downvote US :>

Re: Why should I do production support?

#79

Post by Galanwe » Tue, Sep 22, 2020, 11:57 AM UTC

dtoma wrote: post #57

Galanwe wrote: post #30

Earlier quoted context omitted.

The cure - proper labor law. Seriously

Historically, I don't think we've made much progress in terms of pro-worker labor laws without said workers first having to get together to create bargaining power. By default the power is in the hands of the ruling class, and they have no interest in handing out paid holidays, reasonable working hours, or sick leave.

Well, there is still that old thing called universal election, where basically people can vote for their leader.

If other things are considered more valuable than proper labor law (like, say, building a wall), then I guess voters get what they deserved.

Re: Why should I do production support?

#80

Post by johnbellone » Tue, Sep 22, 2020, 12:01 PM UTC

lostdog wrote: post #35

ed25519FUUU wrote: post #11

Earlier quoted context omitted.

Having skin in the game will make you a better engineer. You’ll get better at the non-sexy things: monitoring, alerting, testing, etc. You better believe a person who is pages for software at 3:00 am has an incentive to make that software more reliable.

But the directors and VP's aren't getting paged at 3am, so the incentives of the organization still go to lower quality software.

If the directors and VPs aren’t getting paged at 3am you need better leadership. The first thing that I did was subscribe to all outages. And I let the operations center to call me anytime if they need help resolving a production issue.