Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why should I do production support?

devenbhooshan.wordpress.com

Post reply on HN
Print view View original

91–100 of 136 posts

Re: Why should I do production support?

#91

Post by taylodl »

danielheath wrote: post #59
crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

Burning out from on-call comes from not being able to fix the underlying causes. I'm happy to hold the pager if I've also got the right to block/rollback deploys until the system is stable - my current job has had two out-of-hours pages in the past year, and we're in the alexa top 10k so it's not like there's no traffic.

Bingo! There's nothing more motivating to "prioritize" fixing a problem than being woken up at 2 a.m. being affected by it. I've worked in places where one team writes the code and another team supports it and it's always the same: the code is absolute garbage. It ends up being too complex and having way too many moving parts, and is impossible to diagnose. I've worked in places where the team who writes it, supports it - 24x7. Their code is always super simple, easy to diagnose, and easy to maintain.

I've been at this for 35 years at many companies and working with many teams and it's always the same: if you want good software then make the team creating it also support it. In every case I've experienced it leads to software requiring little to no support, easy to maintain, and easy to extend. Why? Because nobody wants to get up in the middle of the night or work weekends and moreover, they'd rather be adding features than limping along with existing features.

Re: Why should I do production support?

#92

Post by throwaway0a5e »

lifeisstillgood wrote: post #23
crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

The cure - Unionisation seriously

Police officers have pretty much the strongest unions out there and the junior folks on the force (i.e. exactly the kind of people who get put on call in tech) generally wind up stacking absurd combinations of shifts in order to be paid competitively. Rail workers, another strongly unionized profession, have it no better.

I get that you like unions but just because you have a hammer doesn't make every problem a nail.

Re: Why should I do production support?

#93

Post by robmsmt »

0xbadcafebee wrote: post #90
sdevonoes wrote: post #65

I like to write software but I don't want to be on-call if the software I wrote breaks at 3am in the morning. I do take my job with professionalism, I do write tests for most of it (not 100% coverage, but 100% coverage of the critical parts), I do monitoring (and answer and fix alerts if they happen during working hours) and I don't deploy on Fridays (and don't allow people to deploy on Fridays). My code will crash s…

This attitude almost always turns into the following: On-call: "Hey devs, I'm being woken up at 3AM because your app sucks. Please fix it." Devs: "Sure, no problem." 4 months go by On-call: "These alerts are still coming in at 3AM. Did you fix the issue?" Dev: "We have a lot of work, we can't dedicate all our time to some minor problems, we have a deadline." Next week, Devs are put on-call. The alerts are fixed in tw…

enter devops

Re: Why should I do production support?

#94

Post by nemetroid »

nineteen999 wrote: post #37
dopylitty wrote: post #17

Earlier quoted context omitted.

The point about on-call is really critical and really on-point. If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse. It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the nigh…

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours. > S…

> Two hours callout in the middle of the night, two (paid) hours given back on their next working day

This still sounds cheap to me. I have never worked on-call (and I never plan to), but the exhaustion cost of working two hours in the middle of the night is not equal to two hours of uninterrupted sleep. I would expect to get at least a half day off (paid) for any amount of middle-of-the-night work.

Re: Why should I do production support?

#95

Post by kolla »

machello13 wrote: post #84
nineteen999 wrote: post #37

Earlier quoted context omitted.

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours. > S…

> Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff. Every company should do this but none I've worked at do. To be honest, I just take the makeup time myself.

Well two callout hours in the night should at least be 4 hours paid during day.

Re: Why should I do production support?

#96

Post by sokoloff »

machello13 wrote: post #84
nineteen999 wrote: post #37

Earlier quoted context omitted.

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours. > S…

> Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff. Every company should do this but none I've worked at do. To be honest, I just take the makeup time myself.

Most companies don't watch the clock for their software engineers. If you get called in the middle of night and take the makeup time yourself, does the company give you time in lieu or not? (By outcome, I would say that they do.)

Re: Why should I do production support?

#97

Post by x87678r »

pawelmi wrote: post #34
woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any b…

> just "the way it has always been done" in finance

Twenty years ago for many systems devs could do whatever they wanted in production. There were insider trading scandals and combined with SOX, regulators cracked down on it so now devs have lost at least write access. If you have an old system that relied on knowledgeable devs to fix stuff its a terrible situation where people just quit and no one can support it.

Re: Why should I do production support?

#98

Post by x87678r »

MattGaiser wrote: post #2

People seem to get stuck in support work. That would be my aversion to doing too much of it. At some of my prior workplaces, there have been people who have been so good at support that they never got assigned to do new development work. Engineers far more experienced and senior than I was (or currently am) were dealing with trivial issues as they were good at it while I got the nice greenfield project. They had to q…

This has been a huge problem for me. I've always loved working with customers and dealing with real problems under pressure. So have enjoyed my time on support rota - but quickly L1 guys learn I'm better at support that other team members so everyone contacted me directly. The grumpy devs who pretend they dont know anything about the live system get interesting projects, no interruptions and a much better resume. I've learned to say no. Its also a good reason to move teams as you're not the super experienced guy who has to fix the urgent gnarly support issues.

Re: Why should I do production support?

#99

Post by redis_mlc »

Competent DBAs have previously encountered these items for over 25 years:

> because one of our database vm restarted

Using VMs for production databases is always a problem because of correlated software updates and tail latency. DBAs don't like to share. Just say no.

> we forgot to change the data type of one column from integer to big int(integer overflow)

It's a good idea to dump your schema and grep for the key columns periodically. Then fix and monitor them.

(In MySQL, there's a number of ways to rapidly use up autoinc values, including master-slave replication (esp. Galera) and upsert statements.)

When I go through the RCAs for busy sites and there's no DBA, usually more than half of the outages were caused by mismanaging databases, often something as simple as a missing index. Doubly so for SV unicorns. - Coinbase is one of the most egregious.

Re: Why should I do production support?

#100

Post by runawaybottle »

MattGaiser wrote: post #2

People seem to get stuck in support work. That would be my aversion to doing too much of it. At some of my prior workplaces, there have been people who have been so good at support that they never got assigned to do new development work. Engineers far more experienced and senior than I was (or currently am) were dealing with trivial issues as they were good at it while I got the nice greenfield project. They had to q…

Some people have a switch then turns them into bug fixing demons. I know I get a rush out of fixing prod issues. People notice that, especially when you just jump into foreign code that you never touched and come out with a fix within an hour or two.
Post reply on HN