As a software engineer doing infrastructure work I often find myself working on operational stuff (mostly chasing weird bugs, some on-call, etc.). In my position I am also expected to release features and do development too, but I feel like it's very difficult to focus because of all the operational issues I am dealing with. How are you guys dealing with that sort of work?
Ask HN: How do you deal with operational work as a software engineer?
1–10 of 39 posts
Re: Ask HN: How do you deal with operational work as a software engineer?
#2What issues exactly are you dealing with? You only provided a very vague
description of this "operational stuff" you do and are disturbed with.
Re: Ask HN: How do you deal with operational work as a software engineer?
#3What issues exactly are you dealing with? You only provided a very vague description of this "operational stuff" you do and are disturbed with.
Dealing with production problems, which may be functionality, performance, and reliability related.
Re: Ask HN: How do you deal with operational work as a software engineer?
#4Chasing bugs and being on-call sound like core parts of a software engineer's job, rather than operational work.
That said, some teams at my company are experimenting with having a week-long rotation for "bread box" issues. Those include tending issues/PRs in open source repos, handling bugs as they come in, etc. That frees up the rest of the rest of the team to work on core feature work.
I like to keep a running list of smaller, non-urgent tasks that would otherwise get neglected. When I have a long-running script or need to take a break from another project, I can refer to the list.
Re: Ask HN: How do you deal with operational work as a software engineer?
#5Chasing bugs and being on-call sound like core parts of a software engineer's job, rather than operational work. That said, some teams at my company are experimenting with having a week-long rotation for "bread box" issues. Those include tending issues/PRs in open source repos, handling bugs as they come in, etc. That frees up the rest of the rest of the team to work on core feature work. I like to keep a running lis…
Chasing bugs? Yes. Being on-call? No. Not unless you signed up for that. Too many companies think they can just get Pagerduty going and sign up all their engineering staff for operations duty. This is stupid for a number of reasons least of which is managed services get rid of most of the need for this and it is typically cheaper than developer time.
Do some developers on the team need to think about scale? Yes. Should all the developers be on call because perhaps the company decided to roll it's own infrastructure and someone has to deal with occasional server with full disks? No.
Re: Ask HN: How do you deal with operational work as a software engineer?
#6It depends on the frequency and nature of these issues, but it sounds like you are experiencing technical debt and that your paying for it with slower development speed. Solving the stability issues should take precedence over developing new features.
Is the stuff you have to intervene for under your control or external? If you're relying on outside systems that are flakey then you need make your systems more resilient, things like automatically retrying a few minutes later if some third party service is down and/or being more transnational so you can deal with errors.
Re: Ask HN: How do you deal with operational work as a software engineer?
#7Chasing bugs and being on-call sound like core parts of a software engineer's job, rather than operational work. That said, some teams at my company are experimenting with having a week-long rotation for "bread box" issues. Those include tending issues/PRs in open source repos, handling bugs as they come in, etc. That frees up the rest of the rest of the team to work on core feature work. I like to keep a running lis…
Chasing bugs? Yes. Being on-call? No. Not unless you signed up for that. Too many companies think they can just get Pagerduty going and sign up all their engineering staff for operations duty. This is stupid for a number of reasons least of which is managed services get rid of most of the need for this and it is typically cheaper than developer time. Do some developers on the team need to think about scale? Yes. Shou…
The flipside to this is that being on call forces developers to care about bugs in their code that cause operational headaches instead of just throwing releases with varying degrees of test coverage over the fence to ops. Funny how certain bugs that languished in the background get priority when the dev responsible for that code's phone is the one that rings at 3am instead of some poor schmuck on the ops team.
Re: Ask HN: How do you deal with operational work as a software engineer?
#8Badly. We've lost a couple devs/ops people in the last year, and haven't adequately replaced them. We're stretched way too thin and everyone is getting very burned out.
I haven't done any significant development work in more than six months, just chasing bugs, doing support, and fussing with email and meetings. It blows; I've got to find a different job.
Re: Ask HN: How do you deal with operational work as a software engineer?
#9What issues exactly are you dealing with? You only provided a very vague description of this "operational stuff" you do and are disturbed with.
Dealing with production problems, which may be functionality, performance, and reliability related.
Speaking as an ops person, my first thought is that you have technical or architecture debt. Obviously, big and/or very rapidly growing systems will hit limits and need constant attention, but these days designing most applications to scale is not a problem.
The root cause of many operations issues that I see these days stems from one or more deficiencies in the development process. I don't say "deficiencies in developers": to get safe development at speed, you need a disciplined development process with appropriate feedback mechanisms: unit tests, integration tests, performance tests, static analysis, code review etc. The default state of code is "buggy", because humans are not perfect.
Re: Ask HN: How do you deal with operational work as a software engineer?
#10It's hard to read this and not want to offer help. I don't know if this is the best venue though.