Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you deal with operational work as a software engineer?

news.ycombinator.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Ask HN: How do you deal with operational work as a software engineer?

#31

Post by toomuchtodo »

bradhe wrote: post #24
amriksohata wrote: post #20

Never agree to be on call and if you do, make sure you are being paid double salary as a minimum, all modern science points to working unsociable hours as a massive detriment to your health. Also working Saturdays and Sundays does not make your team more productive, because your staff will be tired the following week, it's a false economy.

> Never agree to be on call and if you do, make sure you are being paid double salary as a minimum Good luck with that.

> Good luck with that.

Agree entirely. The law hasn't caught up requiring compensation for on call/off hours work at the existing salary level (at least in the US).

Moving from devops/infrastructure to security I doubled my total comp (salary + 401k + vacation + health insurance) while reducing the hours per week I work down to ~37 hours (I also get to work remote and never work nights or weekends). If you're in ops/devops/infrastructure, I highly recommend the transition to others, the current demand for competent security professionals is quite literally bananas.

TL;DR If you're in ops, get out of ops (easy) or work someplace that will compensate you appropriately for on call/nights/weekends work (hard).

Re: Ask HN: How do you deal with operational work as a software engineer?

#32

Post by irascible »

twalla wrote: post #7
mottomotto wrote: post #5

Earlier quoted context omitted.

Chasing bugs? Yes. Being on-call? No. Not unless you signed up for that. Too many companies think they can just get Pagerduty going and sign up all their engineering staff for operations duty. This is stupid for a number of reasons least of which is managed services get rid of most of the need for this and it is typically cheaper than developer time. Do some developers on the team need to think about scale? Yes. Shou…

The flipside to this is that being on call forces developers to care about bugs in their code that cause operational headaches instead of just throwing releases with varying degrees of test coverage over the fence to ops. Funny how certain bugs that languished in the background get priority when the dev responsible for that code's phone is the one that rings at 3am instead of some poor schmuck on the ops team.

I've worked at a large mmo company where engineers were randomly assigned pager/ops duty for some weeks. It's a good form of dog fooding,but needs to be done in a way that respects people's existing roles and obligations.. i.e. needs to be scheduled and agreed upon.

Re: Ask HN: How do you deal with operational work as a software engineer?

#33

Post by mottomotto »

rocmcd wrote: post #22
twalla wrote: post #7

Earlier quoted context omitted.

The flipside to this is that being on call forces developers to care about bugs in their code that cause operational headaches instead of just throwing releases with varying degrees of test coverage over the fence to ops. Funny how certain bugs that languished in the background get priority when the dev responsible for that code's phone is the one that rings at 3am instead of some poor schmuck on the ops team.

This exactly. If the developers responsible for the problem (and the fix) aren't feeling the pain of being on-call, then nothing will change and the fallout will be left on support/ops (who will usually find a poorly thought out workaround). Do developers need to be on-call to handle purely ops-related activities (low disk space, high system load, etc)? Absolutely not. Should developers be responsible for their "prod…

But the problem is if you assign a rotating duty to your engineering staff, you as an engineer have no direct impact on how often you will be called due to the half-assed work of other developers. It's a rocky road. Do this too much and your staff will leave. I certainly will. Life is too short.

In short, we're all describing poor management issues. Signing up all the developers for Pagerduty is band aid. So is pushing it all onto operations. In both cases, management is making a choice to avoid dealing with something that requires ongoing effort and time.

Re: Ask HN: How do you deal with operational work as a software engineer?

#34

Post by SirZimzim »

toomuchtodo wrote: post #31
bradhe wrote: post #24

Earlier quoted context omitted.

> Never agree to be on call and if you do, make sure you are being paid double salary as a minimum Good luck with that.

> Good luck with that. Agree entirely. The law hasn't caught up requiring compensation for on call/off hours work at the existing salary level (at least in the US). Moving from devops/infrastructure to security I doubled my total comp (salary + 401k + vacation + health insurance) while reducing the hours per week I work down to ~37 hours (I also get to work remote and never work nights or weekends ). If you're in ops…

Any suggestions for those of us trying to enter the security market from devops?

Re: Ask HN: How do you deal with operational work as a software engineer?

#35

Post by toomuchtodo »

SirZimzim wrote: post #34
toomuchtodo wrote: post #31

Earlier quoted context omitted.

> Good luck with that. Agree entirely. The law hasn't caught up requiring compensation for on call/off hours work at the existing salary level (at least in the US). Moving from devops/infrastructure to security I doubled my total comp (salary + 401k + vacation + health insurance) while reducing the hours per week I work down to ~37 hours (I also get to work remote and never work nights or weekends ). If you're in ops…

Any suggestions for those of us trying to enter the security market from devops?

If you know how to securely design and build AWS environments, you qualify for “cloud security architect” positions based on my interviewing experience. Beyond that, interview for security positions and take note of your gaps to improve on.

Re: Ask HN: How do you deal with operational work as a software engineer?

#36

Post by lin_lin »

nvarsj wrote: post #29
lin_lin wrote: post #18

Earlier quoted context omitted.

On call as a core part? Really? Thankfully I've never worked anywhere with such a "duty", tbh if my current place proposed it I'd be applying for new jobs by lunch time. What's the standard pay for being on-call as a matter of interest?

Every healthy engineering place I've worked at had developers on call. It's called "eating your own dog food". Devs should be responsible for the things they build - it affects the dev culture significantly when your shitty code can wake you up in the middle of the night.

So realistically, is the code going to be fixed at 3 a.m.? Why can't it wait til I'm in the next morning at 8 a.m. for a proper review, triage, priority listing and then fix?

I'm shocked that people would so easily give up their free time really, but to each their own.

How much does it pay extra?

> It's called "eating your own dog food".

No it's not, that's using your own product. Which I do.

Re: Ask HN: How do you deal with operational work as a software engineer?

#37

Post by nvarsj »

lin_lin wrote: post #36
nvarsj wrote: post #29

Earlier quoted context omitted.

Every healthy engineering place I've worked at had developers on call. It's called "eating your own dog food". Devs should be responsible for the things they build - it affects the dev culture significantly when your shitty code can wake you up in the middle of the night.

So realistically, is the code going to be fixed at 3 a.m.? Why can't it wait til I'm in the next morning at 8 a.m. for a proper review, triage, priority listing and then fix? I'm shocked that people would so easily give up their free time really, but to each their own. How much does it pay extra? > It's called "eating your own dog food". No it's not, that's using your own product. Which I do.

> is the code going to be fixed at 3 a.m.?

Yes, if it matters. I can think of dozens of examples. E.g. you provide a payment processor, and you have clients worldwide.

At any significant size, it's going to matter if your service is down for 5 hours. Let's take an extreme example - let's say Google search goes down at 3am PST. Do you think the engineer on call is going to wait 5 hours before "triage, priority listing and then fix"? Are you kidding me?

I don't know what bubble you live in, but in the real world and for many businesses, outages out of hours matter. I'm sure some places they don't (like maybe day trading systems).

If you don't want a call out, build your systems to be resilient to failure, and self-healing.

Re: Ask HN: How do you deal with operational work as a software engineer?

#38

Post by nvarsj »

nvarsj wrote: post #37
lin_lin wrote: post #36

Earlier quoted context omitted.

So realistically, is the code going to be fixed at 3 a.m.? Why can't it wait til I'm in the next morning at 8 a.m. for a proper review, triage, priority listing and then fix? I'm shocked that people would so easily give up their free time really, but to each their own. How much does it pay extra? > It's called "eating your own dog food". No it's not, that's using your own product. Which I do.

> is the code going to be fixed at 3 a.m.? Yes, if it matters. I can think of dozens of examples. E.g. you provide a payment processor, and you have clients worldwide. At any significant size, it's going to matter if your service is down for 5 hours. Let's take an extreme example - let's say Google search goes down at 3am PST. Do you think the engineer on call is going to wait 5 hours before "triage, priority listing…

On the flip side - if your workplace is toxic or bad enough that you aren't allowed to fix systemic issues that cause outages, well then I can see your viewpoint. It's not worth being on call if you can't make things better.

Re: Ask HN: How do you deal with operational work as a software engineer?

#39

Post by lin_lin »

nvarsj wrote: post #37
lin_lin wrote: post #36

Earlier quoted context omitted.

So realistically, is the code going to be fixed at 3 a.m.? Why can't it wait til I'm in the next morning at 8 a.m. for a proper review, triage, priority listing and then fix? I'm shocked that people would so easily give up their free time really, but to each their own. How much does it pay extra? > It's called "eating your own dog food". No it's not, that's using your own product. Which I do.

> is the code going to be fixed at 3 a.m.? Yes, if it matters. I can think of dozens of examples. E.g. you provide a payment processor, and you have clients worldwide. At any significant size, it's going to matter if your service is down for 5 hours. Let's take an extreme example - let's say Google search goes down at 3am PST. Do you think the engineer on call is going to wait 5 hours before "triage, priority listing…

You've given an extreme example, not worth replying to really.

> I don't know what bubble you live in

The "bubble" that the vast majority of devs live in where on-call isn't a thing?

> If you don't want a call out

I won't agree to one, nor sign a contract where it's listed.

Post reply on HN