Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you deal with operational work as a software engineer?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Ask HN: How do you deal with operational work as a software engineer?

#21

Post by scarface74 »

amriksohata wrote: post #20

Never agree to be on call and if you do, make sure you are being paid double salary as a minimum, all modern science points to working unsociable hours as a massive detriment to your health. Also working Saturdays and Sundays does not make your team more productive, because your staff will be tired the following week, it's a false economy.

So if your software needs to run 24 hours and something breaks with your software, how do you avoid being on call?

A developer shouldn't be the first person called, there should be an operations staff but they may have to escalate.

On the other hand, any time that a developer is routinely being called in the middle of the night, there is usually either an issue with the software or the infrastructure not being fault tolerant.

Re: Ask HN: How do you deal with operational work as a software engineer?

#22

Post by rocmcd »

twalla wrote: post #7
mottomotto wrote: post #5

Earlier quoted context omitted.

Chasing bugs? Yes. Being on-call? No. Not unless you signed up for that. Too many companies think they can just get Pagerduty going and sign up all their engineering staff for operations duty. This is stupid for a number of reasons least of which is managed services get rid of most of the need for this and it is typically cheaper than developer time. Do some developers on the team need to think about scale? Yes. Shou…

The flipside to this is that being on call forces developers to care about bugs in their code that cause operational headaches instead of just throwing releases with varying degrees of test coverage over the fence to ops. Funny how certain bugs that languished in the background get priority when the dev responsible for that code's phone is the one that rings at 3am instead of some poor schmuck on the ops team.

This exactly. If the developers responsible for the problem (and the fix) aren't feeling the pain of being on-call, then nothing will change and the fallout will be left on support/ops (who will usually find a poorly thought out workaround).

Do developers need to be on-call to handle purely ops-related activities (low disk space, high system load, etc)? Absolutely not. Should developers be responsible for their "production-ready" code when it breaks? Definitely.

Re: Ask HN: How do you deal with operational work as a software engineer?

#23

Post by Jtsummers »

Identify points to automate. Automate them. Get the automation peer reviewed by the team. Establish testing for the automation. Deploy the automation.

If it's one-offs and not consistent misbehavior that the above can deal with, improve testing infrastructure. If you're unable to hit your feature development schedule, point to the problems in the present system and infrastructure.

Ask your boss for clear priorities: Do they want a stable system, or more features. If the present system is this unstable, then more features will only exarcibate this. If they say they want both, and give them equal priority, ask for a pay raise and search for new jobs.

Re: Ask HN: How do you deal with operational work as a software engineer?

#24

Post by bradhe »

amriksohata wrote: post #20

Never agree to be on call and if you do, make sure you are being paid double salary as a minimum, all modern science points to working unsociable hours as a massive detriment to your health. Also working Saturdays and Sundays does not make your team more productive, because your staff will be tired the following week, it's a false economy.

> Never agree to be on call and if you do, make sure you are being paid double salary as a minimum

Good luck with that.

Re: Ask HN: How do you deal with operational work as a software engineer?

#26

Post by aprdm »

You need a better system in place to prevent bugs from happening.

- Separation between development / staging / production environments.

- Integration tests.

- Service / System Metrics.

- Central logging.

- High availability.

- Alerts.

When you have a solid deployment pipeline things don't usually break. Errors and regressions are caught in the staging part of the deployment pipeline and errors in production can be rolled back automatically (and then you add a integration test for the regression!)

All this devopsy work at my company is done by software engineers with advise from systems engineers. And we do it because neither of the groups want to get called in the weekends :) it has been working really well. Last year we had 0 calls. Before we had this in place things would break in a weekly basis.

You can build all of what I mentioned with OSS like:

- Ansible (deployment)

- Jenkins (ci)

- ELK stack (metrics / logging)

- Zabbix (system metrics)

This system has been serving us, on premises, without much maintenance.

Re: Ask HN: How do you deal with operational work as a software engineer?

#27

Post by [deleted] »

amriksohata wrote: post #20

Never agree to be on call and if you do, make sure you are being paid double salary as a minimum, all modern science points to working unsociable hours as a massive detriment to your health. Also working Saturdays and Sundays does not make your team more productive, because your staff will be tired the following week, it's a false economy.

[deleted]

Re: Ask HN: How do you deal with operational work as a software engineer?

#28

Post by amriksohata »

scarface74 wrote: post #21
amriksohata wrote: post #20

Never agree to be on call and if you do, make sure you are being paid double salary as a minimum, all modern science points to working unsociable hours as a massive detriment to your health. Also working Saturdays and Sundays does not make your team more productive, because your staff will be tired the following week, it's a false economy.

So if your software needs to run 24 hours and something breaks with your software, how do you avoid being on call? A developer shouldn't be the first person called, there should be an operations staff but they may have to escalate. On the other hand, any time that a developer is routinely being called in the middle of the night, there is usually either an issue with the software or the infrastructure not being fault…

In the UK there are laws you can opt out of being asked to work more than a certain amount of hours. They company should have an out of hours plan but most experienced developers will know very few things get resolved in the middle hours of the night, things need testing, reviewing and sometimes the solution is not simple, it is better like you said to have ops staff that gather data and then pass it on when devs are in fresh, however if you have, say, a big international sale which is happening in another timezone then why not just pay staff as a one off to be around?

Re: Ask HN: How do you deal with operational work as a software engineer?

#29

Post by nvarsj »

lin_lin wrote: post #18
kaikai wrote: post #4

Chasing bugs and being on-call sound like core parts of a software engineer's job, rather than operational work. That said, some teams at my company are experimenting with having a week-long rotation for "bread box" issues. Those include tending issues/PRs in open source repos, handling bugs as they come in, etc. That frees up the rest of the rest of the team to work on core feature work. I like to keep a running lis…

On call as a core part? Really? Thankfully I've never worked anywhere with such a "duty", tbh if my current place proposed it I'd be applying for new jobs by lunch time. What's the standard pay for being on-call as a matter of interest?

Every healthy engineering place I've worked at had developers on call. It's called "eating your own dog food". Devs should be responsible for the things they build - it affects the dev culture significantly when your shitty code can wake you up in the middle of the night.

Re: Ask HN: How do you deal with operational work as a software engineer?

#30

Post by oceanghost »

twalla wrote: post #7
mottomotto wrote: post #5

Earlier quoted context omitted.

Chasing bugs? Yes. Being on-call? No. Not unless you signed up for that. Too many companies think they can just get Pagerduty going and sign up all their engineering staff for operations duty. This is stupid for a number of reasons least of which is managed services get rid of most of the need for this and it is typically cheaper than developer time. Do some developers on the team need to think about scale? Yes. Shou…

The flipside to this is that being on call forces developers to care about bugs in their code that cause operational headaches instead of just throwing releases with varying degrees of test coverage over the fence to ops. Funny how certain bugs that languished in the background get priority when the dev responsible for that code's phone is the one that rings at 3am instead of some poor schmuck on the ops team.

This works the other direction as well.

Managements risk-taking is essentially guaranteed by free employee overtime.

Post reply on HN