Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why should I do production support?

devenbhooshan.wordpress.com

Post reply on HN
Print view View original

121–130 of 136 posts

Re: Why should I do production support?

#121

Post by fatnoah »

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

>The "typical" on-call - where when you are on-call you are magically on-call 24x7.

I ran the Engineering org for a startup and we had a small, 3 person Ops team that handled initial triage of events. About 75% of these issues were Engineering-related. My solution was to a) create an on-call rotation for Engineering and b) allow the Engineers to prioritize reliability work.

It sounds like a no-brainer, but I had to fight with the rest of the exec team to allow b) to happen, since it came at the expense of the product roadmap. I eventually won the fight and our nightly on-call volume went from 1-2 incidents per day to 1-2 every few months.

Vindication came about a year later when we were acquired by a large company. As part of the due diligence process (including 18 hours with me going over technical details in front of 30 senior folks from the acquiring company) we got major kudos for having a level of reliability that far exceeded what they typically saw for a company our size.

Re: Why should I do production support?

#122

Post by fatnoah »

nineteen999 wrote: post #37
dopylitty wrote: post #17

Earlier quoted context omitted.

The point about on-call is really critical and really on-point. If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse. It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the nigh…

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours. > S…

>Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff.

I didn't note this in my post above, but I always gave time-in-lieu for any late night activity. However, the thing that REALLY worked best was allowing the Engineers to prioritize reliability. I had to fight to make it happen, but going from nightly to every couple months volumes was worth it.

Re: Why should I do production support?

#123

Post by greesil »

goatinaboat wrote: post #44
greesil wrote: post #4

At some large tech companies, production support as a software engineer does not seem to be a path to promotion, unless you are a junior level engineer. And yet, solving some of the bugs encountered in production environments, especially with a heterogeneous set of users, requires expert level knowledge of a particular software library. It is probably a great way to coast, so I hear.

It is probably a great way to coast, so I hear. Or to stagnate, depending on how you look at it

One part of your life stagnates while the other parts move forward.

Re: Why should I do production support?

#124

Post by gertrunde »

detaro wrote: post #62
zeckalpha wrote: post #61

Earlier quoted context omitted.

If the engineers aren’t oncall, who is? Is it okay to exploit non-engineers? If anything, it is less exploitative to have those who are empowered to improve their situation oncall.

Given that the parent comment is clear about the problem being the 24x7 expectation: Someone paid to work or be available that shift , engineer or not?

There are too many variables and factors involved really.

Is someone covering the full 24 hours, or is it just out of hours?

Are they under a one week in X rota, or expected to be permanently available?

Are they expected to cover anything/everything, or are they just the escalation point for their specialist area?

What other support is available? (i.e. if the shit really hits the fan, are you left to deal with it alone?)

On average, how many times are callouts expected? There's a big difference between half a dozen times a week and half a dozen times a year.

How are the extra duties remunerated/compensated? Is there time off in lieu?

There's a massive spectrum there ranging from hugely unpleasant and not worth the money to not particularly onerous and helpful extra cash/time off.

Re: Why should I do production support?

#125

Post by jjirsa »

ocdtrekkie wrote: post #107
TuringNYC wrote: post #26

Earlier quoted context omitted.

Dont most developers do level III support? Ultimately if the first and second lines of defense cannot solve the problem, from everything I've seen, it goes to an Engineer. If it is your product, i'd assume it comes to you. This has been the case at almost every company I've been at.

Don't dismiss the value of tier 1 and 2 support requests. Maybe 30% of your tier 1 requests are some confusion that support knows how to alleviate, but an engineer could make the product more straightforward to eliminate those support issues entirely. You aren't looking for the hardest problems, you're looking for the problems your users hit the most that an engineer could reduce in the product.

> You aren't looking for the hardest problems, you're looking for the problems your users hit the most that an engineer could reduce in the product.

This is the hard part of software.

Re: Why should I do production support?

#126

Post by machello13 »

sokoloff wrote: post #96
machello13 wrote: post #84

Earlier quoted context omitted.

> Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff. Every company should do this but none I've worked at do. To be honest, I just take the makeup time myself.

Most companies don't watch the clock for their software engineers. If you get called in the middle of night and take the makeup time yourself, does the company give you time in lieu or not? (By outcome, I would say that they do.)

That's fair. It would be nice if it was an explicit policy though. Otherwise I feel like the company is just exploiting engineers who may not know any better.

Re: Why should I do production support?

#127

Post by peterwwillis »

cabraca wrote: post #104
0xbadcafebee wrote: post #90

Earlier quoted context omitted.

This attitude almost always turns into the following: On-call: "Hey devs, I'm being woken up at 3AM because your app sucks. Please fix it." Devs: "Sure, no problem." 4 months go by On-call: "These alerts are still coming in at 3AM. Did you fix the issue?" Dev: "We have a lot of work, we can't dedicate all our time to some minor problems, we have a deadline." Next week, Devs are put on-call. The alerts are fixed in tw…

Putting the blame on the devs is to easy. There are a bunch of management layers between the on-call team and the devs in your example. If management does not prioritize those alerts, its not the fault of the devs and its the wrong to put punish the devs for it. > Honestly, the whole attitude of not wanting to work more than 8 hours is privilege. Most of the rest of the world works long hours. As a dev, you get a goo…

Say a plumber installs a new sink in a house. A year later, a weak joint fails and springs a leak, and the plumber is called to fix it. They could ignore the error, charge the person for the repair, and go on with their day. Or they could accept that it failed due to their own mistake and not charge the person.

Now let's say the plumber works for a contractor. The contractor tells the plumber not to tell the customer about the mistake, because it would make the contractor look bad. The plumber can choose to own up to their mistake to the customer and right the wrong, or they can do what the contractor wants and charge the customer.

On the one hand, the plumber might decide to charge the customer. They keep their job, the contractor makes money. On the other hand, maybe the customer is poor and can't really afford the repair. If it doesn't get fixed, the customer'll have to deal with the brokenness themselves, even though the plumber knows they caused it. But then again, maybe the plumber is broke and really needs this subcontracting gig.

There is no simple answer there. But I think that in the context of software development, in most cases, the answer is simple. Most of us are fortunate enough to have the extra time and money to spend fixing our own bugs, regardless of what we're told to do during the regular 9-to-5. When we have the opportunity to take responsibility for our actions, we oughta.

Re: Why should I do production support?

#128

Post by zeckalpha »

detaro wrote: post #62
zeckalpha wrote: post #61

Earlier quoted context omitted.

If the engineers aren’t oncall, who is? Is it okay to exploit non-engineers? If anything, it is less exploitative to have those who are empowered to improve their situation oncall.

Given that the parent comment is clear about the problem being the 24x7 expectation: Someone paid to work or be available that shift , engineer or not?

No, they said “normal office hours” are for engineers. Someone needs to take the other shifts.

Re: Why should I do production support?

#129

Post by zeckalpha »

magicalhippo wrote: post #71
zeckalpha wrote: post #61

Earlier quoted context omitted.

If the engineers aren’t oncall, who is? Is it okay to exploit non-engineers? If anything, it is less exploitative to have those who are empowered to improve their situation oncall.

Our support folks handle the on-call support. They do one week each on rotation, they get some extra compensation and the following friday off. If it's a serious issue they can't handle they might wake up one of us programmers, but usually they can find some temporary fix or workaround until the next morning.

Do permanent fixes get applied or do the mitigations just get re-applied again and again?

Are the support folks considered engineers?

Re: Why should I do production support?

#130

Post by woutr_be »

pawelmi wrote: post #119
woutr_be wrote: post #50

Earlier quoted context omitted.

> BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Not only that, but once that happens, regulators will come in, and everybody involved can be held liable. Not only will the bank be fined, but depending on how bad your fuck up was, you'll probably end up losing your job and might face further penalties. So in the interest of eve…

Thanks for answers in this thread. I didn't really mean to have YOLO-type random access to production. I was hoping there are ways in between to bridge that gap between dev and ops in those systems, similarly how it has been done eg with SRE in more relaxed security applications. I was hoping for some solutions on the spectrum are adopted more, like mentioned cetralized logs stripped of private data or granting tempo…

At least where I work, every team can essentially decide what they do, as long as they follow a few basic guidelines. So newer projects usually have centralised logging and automated deployments. But sadly there’s still a wall between development environments and production, for good reason I think. Not everyone should have access to production data, so only a limited amount of people have access. Data is of course anonymised when send over.

But yeah, some legacy systems could be 5 years old, and that’s a long time in tech.

You’re right on the visibility part, but sadly that’s an organisational issue, you need higher ups to change this.

Post reply on HN