Why should I do production support? • Hacker News Board

Re: Why should I do production support?

#41

Post by goatinaboat » Tue, Sep 22, 2020, 6:05 AM UTC

pawelmi wrote: post #34

woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any b…

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software

They can, they just can’t have direct access to live systems due to separation of duties. But there are methods for dealing with this, like centralised logging so a developer never needs to see the original log file on the problematic box.

Re: Why should I do production support?

#42

Post by woutr_be » Tue, Sep 22, 2020, 6:05 AM UTC

pawelmi wrote: post #34

woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any b…

Part of it probably stems from that it's always been done this way, and the processes haven't evolved over time. Another part is also to protect the bank against rogue employees, it wouldn't be the first time a developer made changes against a production database.

You're right that production support has access to those systems, and could potentially make changes and install different binaries, but the amount of people that can do that is extremely limited. Every change also requires a change request that needs several approvals, to request data you need another data request.

Re: Why should I do production support?

#43

Post by Ma8ee » Tue, Sep 22, 2020, 6:06 AM UTC

Galanwe wrote: post #30

lifeisstillgood wrote: post #23

Earlier quoted context omitted.

The cure - Unionisation seriously

The cure - proper labor law. Seriously

Which is one of the things that Unions are good at pushing for. Who do you think create laws? A single guy walking up to his Member of Congress.

Re: Why should I do production support?

#44

Post by goatinaboat » Tue, Sep 22, 2020, 6:06 AM UTC

greesil wrote: post #4

At some large tech companies, production support as a software engineer does not seem to be a path to promotion, unless you are a junior level engineer. And yet, solving some of the bugs encountered in production environments, especially with a heterogeneous set of users, requires expert level knowledge of a particular software library. It is probably a great way to coast, so I hear.

It is probably a great way to coast, so I hear.

Or to stagnate, depending on how you look at it

Re: Why should I do production support?

#45

Post by hinkley » Tue, Sep 22, 2020, 6:07 AM UTC

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

It’s usually a case if taking responsibility for things you have no power over.

When they ask it’s usually after a giant hole has been dug, and patterns have been set. If you knew from the outset that you would be the support team then you’d have prioritized some other tickets. You’d have increased the estimates on others. You would have refused to work on these three, you would have argued vigorously about these four decisions, and you would insisted your boss fire “That Guy” months ago because his code is garbage and his only real skill is articulate deflection.

This group of folks wants several somethings for nothing. One of them is labor, another is somewhere to assign blame. They are grooming you for failure and we all deserve better.

Re: Why should I do production support?

#46

Post by woutr_be » Tue, Sep 22, 2020, 6:07 AM UTC

goatinaboat wrote: post #41

pawelmi wrote: post #34

Earlier quoted context omitted.

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any b…

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software They can, they just can’t have direct access to live systems due to separation of duties…

> But there are methods for dealing with this, like centralised logging so a developer never needs to see the original log file on the problematic box.

Indeed, most of our newer services have been doing that, so developers have direct access to logs, which makes our lives a bit easier.

But sadly a lot of systems our outdated, and nobody wants to invest time and money into implementing things there.

Re: Why should I do production support?

#47

Post by HenryBemis » Tue, Sep 22, 2020, 6:09 AM UTC

pawelmi wrote: post #34

woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any b…

Seems like we were writing in parallel.

"We don't trust" a dev. The change management processes demand the existence of 1) Dev, 2) Librarian (we used to call them that)(that would review and transfer the code, or review and compile the code), 3) the prod sys admin.

Some orgs may have a slightly different setup, but in some form or another, but (these general) rules apply.

Today with tools like CyberArk it is easier to grant temporarily privileged access to a dev for production support, we also got the tools to trace/monitor/record access, so it makes the process auditor-friendly.

Re: Why should I do production support?

#48

Post by HenryBemis » Tue, Sep 22, 2020, 6:13 AM UTC

woutr_be wrote: post #42

pawelmi wrote: post #34

Earlier quoted context omitted.

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any b…

Part of it probably stems from that it's always been done this way, and the processes haven't evolved over time. Another part is also to protect the bank against rogue employees, it wouldn't be the first time a developer made changes against a production database. You're right that production support has access to those systems, and could potentially make changes and install different binaries, but the amount of peop…

Adding to that (because only when I use this example people understand clearly):

CompanyA is using ITS OWN assets, funds, IP, etc. you own it, you can burn to the ground.

BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake".

Edit: similar expectations are in publicly traded companies (aka companies where they use OUR money - we give them our cash and they give us stocks). This is why external auditors (e.g. Big4) do not like when they see "poor change management processes", such as inconsistent SoD.

Re: Why should I do production support?

#49

Post by hinkley » Tue, Sep 22, 2020, 6:15 AM UTC

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

> "do you even realise what a service to humanity you are doing!".

I don’t know who needs to hear this besides me twenty years ago, but if you want to do charity then go home at 6pm and volunteer at a real charity. Don’t do it for a wannabe robber baron who will not share with you. Don’t do it for someone where even an emotional payoff is years away or may never come.

Find something else you care about and help some people just because. Not because you’re getting under-paid and over-guilted to do it.

Re: Why should I do production support?

#50

Post by woutr_be » Tue, Sep 22, 2020, 6:17 AM UTC

HenryBemis wrote: post #48

woutr_be wrote: post #42

Earlier quoted context omitted.

Part of it probably stems from that it's always been done this way, and the processes haven't evolved over time. Another part is also to protect the bank against rogue employees, it wouldn't be the first time a developer made changes against a production database. You're right that production support has access to those systems, and could potentially make changes and install different binaries, but the amount of peop…

Adding to that (because only when I use this example people understand clearly): CompanyA is using ITS OWN assets, funds, IP, etc. you own it, you can burn to the ground. BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Edit: similar expectations are in publicly traded companies (aka companies where they use OUR money - we give t…

> BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake".

Not only that, but once that happens, regulators will come in, and everybody involved can be held liable. Not only will the bank be fined, but depending on how bad your fuck up was, you'll probably end up losing your job and might face further penalties.

So in the interest of everyone, it's best to just avoid it all together.