Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why should I do production support?

devenbhooshan.wordpress.com

Post reply on HN
Print view View original

111–120 of 136 posts

Re: Why should I do production support?

#111

Post by user5994461 »

kevinmchugh wrote: post #14
MattGaiser wrote: post #10

Earlier quoted context omitted.

The engineers at the organization I just departed (at least the ones in the support rotation which did not include me) got paid in both money and extra time off for their time spent on support tasks.

Yeah, I had a job where only certain teams had on call rotations. Anyone in those rotations got an extra half day off per week on call. I've also seen folks spotted extra time off for really gnarly oncall shifts. Folks should push to have such accommodations standardized.

A half day off for doing 7 full days of extra work. It's peanuts. Better not do the rotation and have your 2 days of week end. (I assume they got zero extra pay and got called occasionally).

Re: Why should I do production support?

#112

Post by stronglikedan »

0xbadcafebee wrote: post #90
sdevonoes wrote: post #65

I like to write software but I don't want to be on-call if the software I wrote breaks at 3am in the morning. I do take my job with professionalism, I do write tests for most of it (not 100% coverage, but 100% coverage of the critical parts), I do monitoring (and answer and fix alerts if they happen during working hours) and I don't deploy on Fridays (and don't allow people to deploy on Fridays). My code will crash s…

This attitude almost always turns into the following: On-call: "Hey devs, I'm being woken up at 3AM because your app sucks. Please fix it." Devs: "Sure, no problem." 4 months go by On-call: "These alerts are still coming in at 3AM. Did you fix the issue?" Dev: "We have a lot of work, we can't dedicate all our time to some minor problems, we have a deadline." Next week, Devs are put on-call. The alerts are fixed in tw…

> Most of the rest of the world works long hours.

Most of the world works labor jobs, and studies have shown that the body can work longer than the mind without burnout.

Re: Why should I do production support?

#113

Post by [deleted] »

ocdtrekkie wrote: post #12
ed25519FUUU wrote: post #11

Earlier quoted context omitted.

Having skin in the game will make you a better engineer. You’ll get better at the non-sexy things: monitoring, alerting, testing, etc. You better believe a person who is pages for software at 3:00 am has an incentive to make that software more reliable.

I'm not even talking 24/7 pager work. But just tackling some support tickets as part of your job so you see where people are having issues with your product. Too often I see BigCorp development teams seeming blatantly oblivious to where their pain points are, and it's because they aren't forcing their developers to do support. They're pushing code, but they aren't pushing code that solves real problems for people.

[deleted]

Re: Why should I do production support?

#114

Post by kevinmchugh »

user5994461 wrote: post #111
kevinmchugh wrote: post #14

Earlier quoted context omitted.

Yeah, I had a job where only certain teams had on call rotations. Anyone in those rotations got an extra half day off per week on call. I've also seen folks spotted extra time off for really gnarly oncall shifts. Folks should push to have such accommodations standardized.

A half day off for doing 7 full days of extra work. It's peanuts. Better not do the rotation and have your 2 days of week end. (I assume they got zero extra pay and got called occasionally).

On call shifts were very quiet and typically didn't involve any pages or interruptions. It was certainly not even a full day of extra work typically.

Re: Why should I do production support?

#115

Post by comeonseriously »

g051051 wrote: post #82

Our company is trying to move to "devops" and having dev teams on pager duty, doing manual reporting, etc. They seem surprised at the amount of pushback from the devs.

On the one hand, a lot of devs see production support as beneath them. On the other I think managers seem to thing they'll get a net productivity increase doing this, but there's not. Development gets much harder when you have to context shift several times per day. That being said, I do feel devs should do prod support. It gives them a better feel for their apps they build and how they're used and where the customer pain points are. But, I also feel there should be layers of support below the devs; devs should only get the ticket when they're the only ones left who can figure it out.

Re: Why should I do production support?

#116

Post by woutr_be »

arminiusreturns wrote: post #108
woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

>However, production support teams don’t have a real understanding of our application and how it’s build. This reeks of bad documentation to me (which finance is notorious for). If a dev has to be on to support normal prod ops thats largely due to errors in both documentation and often in poor tooling. Sometimes those errors aren't as much the devs fault because of management decisions, usually related to understaffi…

You're not wrong, the entire thing is because of poor management decisions and poor processes. I don't really shit on specific people, more on the entire process.

> This reeks of bad documentation to me

Not necessarily, you can document your entire application, but production support only looks at the logs, and does a data extract based on what they see. It would be far more beneficial if you had someone who has a clear understanding of the application so that they can help with debugging and actually solving the problem.

At the end of the day, production support are teams who help with 10-20 applications, it's impossible for them to truly understand specific applications. They receive a bug report from the business, investigate and extract logs, then pass it to the relevant development teams. If you need extra info, well though luck, you can reply to the ticket and wait for it to be picked up again. It's no surprise companies like this move so slow.

Re: Why should I do production support?

#117

Post by woutr_be »

user5994461 wrote: post #102
woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

Also in finance with dedicated support teams. Our support was great, some of our guys in Asia were outright fantastic at debugging. To be fair, being great wasn't enough, their job was only possible because the company had unified tooling. A single deployment solution that was deploying near 1M tasks a day in the company, allowing all employees to lookup what is running where and see logs. This made me appreciate jus…

We still don't have a single deployment solution, we don't even have single hosting solution. We can choose between 3 different cloud providers, or dedicated servers. (They're pushing for cloud now)

The thing is, this all is pushed down from management. In my previous project, we tried to automate as much as possible, but at the end of the day, our production support still wanted to deploy manually. Our business still wanted to see manual end-to-end tests with screenshots.

Then there's also different regulations in certain countries where you need to host your application and database in the country itself, so that's another solution.

Working in finance can be a real eye opener sometimes.

Re: Why should I do production support?

#118

Post by g051051 »

comeonseriously wrote: post #115
g051051 wrote: post #82

Our company is trying to move to "devops" and having dev teams on pager duty, doing manual reporting, etc. They seem surprised at the amount of pushback from the devs.

On the one hand, a lot of devs see production support as beneath them. On the other I think managers seem to thing they'll get a net productivity increase doing this, but there's not. Development gets much harder when you have to context shift several times per day. That being said, I do feel devs should do prod support. It gives them a better feel for their apps they build and how they're used and where the customer…

It's not "beneath" them, but it's an entirely different skill set. If you want me involved, approach me like a partner and we can work together, don't try to suddenly give me a job I can't do.

Re: Why should I do production support?

#119

Post by pawelmi »

woutr_be wrote: post #50
HenryBemis wrote: post #48

Earlier quoted context omitted.

Adding to that (because only when I use this example people understand clearly): CompanyA is using ITS OWN assets, funds, IP, etc. you own it, you can burn to the ground. BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Edit: similar expectations are in publicly traded companies (aka companies where they use OUR money - we give t…

> BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Not only that, but once that happens, regulators will come in, and everybody involved can be held liable. Not only will the bank be fined, but depending on how bad your fuck up was, you'll probably end up losing your job and might face further penalties. So in the interest of eve…

Thanks for answers in this thread. I didn't really mean to have YOLO-type random access to production. I was hoping there are ways in between to bridge that gap between dev and ops in those systems, similarly how it has been done eg with SRE in more relaxed security applications. I was hoping for some solutions on the spectrum are adopted more, like mentioned cetralized logs stripped of private data or granting temporary audited access. But it seems with legacy systems this is much harder to implements. I believe there is an optimum balance where actually fewer mistakes could be made if both people developing and operating te system had more visibility into each other field. As for willful fraud attempts, well you can't rule out devs would do it, so of course there should be various barriers preventing that and proper change management, but, my sampling bias aside, when I look at some recent scandals in finance, take eg Wirecard as the last one, there is more often higher management involved than devs.

Re: Why should I do production support?

#120

Post by nineteen999 »

nemetroid wrote: post #94
nineteen999 wrote: post #37

Earlier quoted context omitted.

> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours. > S…

> Two hours callout in the middle of the night, two (paid) hours given back on their next working day This still sounds cheap to me. I have never worked on-call (and I never plan to), but the exhaustion cost of working two hours in the middle of the night is not equal to two hours of uninterrupted sleep. I would expect to get at least a half day off (paid) for any amount of middle-of-the-night work.

The thousand bucks extra they make a week whether they get called or not makes up for it. It's not like I clock watch my team anyway, if they took an extra hour here or there I don't bat an eyelid, they're all hard workers and they get the job done without complaining.
Post reply on HN