Why should I do production support? • Hacker News Board

Re: Why should I do production support?

#101

Post by hanniabu » Tue, Sep 22, 2020, 1:40 PM UTC

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

You're talking as if they care about the exploitation. It's done purposely. They simply don't care when it means greater profits. They also believe there's an endless line of devs they can burn out and throw away.

Re: Why should I do production support?

#102

Post by user5994461 » Tue, Sep 22, 2020, 1:43 PM UTC

woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

Also in finance with dedicated support teams. Our support was great, some of our guys in Asia were outright fantastic at debugging.

To be fair, being great wasn't enough, their job was only possible because the company had unified tooling. A single deployment solution that was deploying near 1M tasks a day in the company, allowing all employees to lookup what is running where and see logs.

This made me appreciate just how useful it is to have both dedicated support AND unified tooling. The average company couldn't benefit from having folks on rota because it's impossible to figure out where anything is running.

Re: Why should I do production support?

#103

Post by comeonseriously » Tue, Sep 22, 2020, 1:46 PM UTC

Engineers absolutely should do prod support. But... There should be layers below. It should only come to engineering when nobody else can figure it out.

Re: Why should I do production support?

#104

Post by cabraca » Tue, Sep 22, 2020, 1:46 PM UTC

0xbadcafebee wrote: post #90

sdevonoes wrote: post #65

I like to write software but I don't want to be on-call if the software I wrote breaks at 3am in the morning. I do take my job with professionalism, I do write tests for most of it (not 100% coverage, but 100% coverage of the critical parts), I do monitoring (and answer and fix alerts if they happen during working hours) and I don't deploy on Fridays (and don't allow people to deploy on Fridays). My code will crash s…

This attitude almost always turns into the following: On-call: "Hey devs, I'm being woken up at 3AM because your app sucks. Please fix it." Devs: "Sure, no problem." 4 months go by On-call: "These alerts are still coming in at 3AM. Did you fix the issue?" Dev: "We have a lot of work, we can't dedicate all our time to some minor problems, we have a deadline." Next week, Devs are put on-call. The alerts are fixed in tw…

Putting the blame on the devs is to easy. There are a bunch of management layers between the on-call team and the devs in your example. If management does not prioritize those alerts, its not the fault of the devs and its the wrong to put punish the devs for it.

> Honestly, the whole attitude of not wanting to work more than 8 hours is privilege. Most of the rest of the world works long hours. As a dev, you get a good salary and a job you don't have to break your body to do. The least you can do is be completely responsible for your own code.

unless i signed a contract that states i will do on-call, i'm not gonna do on-call. I doesn't matter how long the rest of the world works.

Re: Why should I do production support?

#105

Post by mjayhn » Tue, Sep 22, 2020, 1:54 PM UTC

danielheath wrote: post #59

crossroadsguy wrote: post #9

You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking. Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem. The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your s…

Burning out from on-call comes from not being able to fix the underlying causes. I'm happy to hold the pager if I've also got the right to block/rollback deploys until the system is stable - my current job has had two out-of-hours pages in the past year, and we're in the alexa top 10k so it's not like there's no traffic.

I'm in ops and this is a huge, huge headache for those of us on our end at companies where we're not given Google SRE powers to control release trajectory based on failure budgets of some sort.

I've worked at too many places that had no SWEs on-call for the on-call alerts that I get, which the vast majority of the time involves throwing a bandaid (as in redirecting traffic, etc) in front of an internal bug that I hope eventually gets fixed once the RFO/etc has been submitted before it hits my NEXT on-call rotation or my poor coworkers.

Without SWEs on my rotation they don't understand the immediacy. They aren't the ones getting their Christmas week interrupted every 4 hours while ops keeps the house running. In Ops having your entire day ruined by various on-call alerts usually feels like you're working without any breaks and nobody even cares.

Anyone want a bad golang developer, wannabe ex-ops person who knows a lot about platform reliability and o11y and wants to focus on the golang end finally? I'll make your teams automation and o11y purr no matter where it is (bm, cloud, global pops, serverless..)..

Re: Why should I do production support?

#106

Post by jasonlotito » Tue, Sep 22, 2020, 1:55 PM UTC

MattGaiser wrote: post #2

People seem to get stuck in support work. That would be my aversion to doing too much of it. At some of my prior workplaces, there have been people who have been so good at support that they never got assigned to do new development work. Engineers far more experienced and senior than I was (or currently am) were dealing with trivial issues as they were good at it while I got the nice greenfield project. They had to q…

Of course, if they were the ones who originally built those products, they should support them. Why should people that create software that requires so much support they quit over it be entrusted with yet another greenfield project without fixing the stuff they built in the first place?

Re: Why should I do production support?

#107

Post by ocdtrekkie » Tue, Sep 22, 2020, 1:55 PM UTC

TuringNYC wrote: post #26

ocdtrekkie wrote: post #7

If you aren't doing production support, you don't actually know your product. You aren't connected to the pain points your users experience and you miss what is, to your support team and your customers, the glaringly obvious. I would argue all developers should be required to do some support work.

Dont most developers do level III support? Ultimately if the first and second lines of defense cannot solve the problem, from everything I've seen, it goes to an Engineer. If it is your product, i'd assume it comes to you. This has been the case at almost every company I've been at.

Don't dismiss the value of tier 1 and 2 support requests. Maybe 30% of your tier 1 requests are some confusion that support knows how to alleviate, but an engineer could make the product more straightforward to eliminate those support issues entirely.

You aren't looking for the hardest problems, you're looking for the problems your users hit the most that an engineer could reduce in the product.

Re: Why should I do production support?

#108

Post by arminiusreturns » Tue, Sep 22, 2020, 2:01 PM UTC

woutr_be wrote: post #6

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly i…

>However, production support teams don’t have a real understanding of our application and how it’s build.

This reeks of bad documentation to me (which finance is notorious for). If a dev has to be on to support normal prod ops thats largely due to errors in both documentation and often in poor tooling. Sometimes those errors aren't as much the devs fault because of management decisions, usually related to understaffing, but I hate how prod support gets shit on so often for failing to fix an issue when it's not really their fault.

Re: Why should I do production support?

#109

Post by user5994461 » Tue, Sep 22, 2020, 2:02 PM UTC

seanwilson wrote: post #28

Another pro for consultancy work: you can chose contracts that that don't require you to be on call for production problems so this isn't forced upon you. I'm not saying you wouldn't learn from working on production, but whether it's worth the stress is another question. In terms of software development, it's hard to think of a worse feeling than when you do a production deploy, you hit refresh on the website or what…

Consultants charge 200% for out of hours work, if not more. I don't think they mind working extra hours.

They're never offered extra work though. Companies are always willing to wait for Monday when they are asked to put money on the table.

Re: Why should I do production support?

#110

Post by ocdtrekkie » Tue, Sep 22, 2020, 2:02 PM UTC

octodog wrote: post #33

jjirsa wrote: post #27

Earlier quoted context omitted.

Why should a company hire two lines of support to shield an engineer from bugs they introduce?

1. Not every support request is the result of a bug. 2. Not every bug is directly related to the product owner/team. Bugs can be introduced from different teams/processes. 3. Requiring senior engineers to do L1 and L2 support is likely a misallocstion of resources when less senior people can handle those issues.

1. The best team I've ever worked with considered every support request to be a bug. Is the user confused? At minimum, that's a documentation bug. It's also possible the product itself should be doing a better job stepping the user through a process or self-diagnosing.

3. https://news.ycombinator.com/item?id=24554399