Ask HN: Does your on-call rotation suck? Can I join it?

I assume it's part of the pay. You can't be a firefighter or a cop and then complain that there's night shifts. I've had nearly 4 years of it at a payment gateway and IIRC only one time was there something that had to be solved that night. When it happened, it was sort of my fault anyway; a good deal of the problems are (should be?) within the control of the people being on-call. And I think companies like payment gateways and cloud services which need people active at all times are also far more tolerant of things like spending a week reviewing a PR and such, so the frequency of downtime is lower even if the impact is much higher.

Though I'd agree it's a staffing issue. 5 people in a cycle is fine. If you had a concert or something that week, just swap places with a colleague. When we reduced it to 2 people, it was not cool to spend half your time on-call.

There's also policies like don't release on Fridays, don't release on a vacation week. If there's a tool for it, it would be flagging these behaviors. Unfortunately, we can't really control when partners go down.

I strongly believe whoever is on-call should have free reign to modify anything about their ops environment. The pitch for management is that while in the short term it might take time away from project work, eventually the reduction in interruptions will result in higher productivity.

p.s. Knew I recognised the name - loved following development of Planimeter/Grid a while ago!

I'm certainly not at liberty to invite a random to cover on-call shifts for us, but here's some anecdata about things I've witness that made on-call suck.

We began with free food delivery over the weekend, and the expectation that you'd take a day off the next week ("unlimited" PTO policy). Eventually they stopped letting us do that and now the "unlimited" in our PTO policy has an invisible limit, so you can't actually do that without it counting towards the invisible limit on your unlimited PTO for the year.

Our monitoring and alerting is unusably noisy. Deviance is fully normalized. All our postmortems typically have a section stating that alerts were issued, but ignored until customers began complaining. Attempts to cut the noise down to a sane level have all been defeated by the ever present pressure to feature factory. TBF this is mostly an engineering self-own and I feel partially responsible for this outcome.

The on-call engineer does a shocking amount of manual labor to paper over bugs in the product and un-stick users who fall through the (many) cracks. It is effectively a T3 tech support rotation. We've taken steps to tone it down to mere triage and channel this into pressure against offending teams' timelines, but there's a huge amount of silent cultural resistance and no one is being held accountable when a feature increases support load. I suspect this issue alone would make most bigtech engineers quit.

For the (many) issues that require manual intervention, the on-call engineer cannot actually do anything unless 2 other engineers sign off on a PR (either to run a SQL query or to deploy some tool or bugfix to resolve the problem).

This is more specific to the product I work on, but the sheer amount of 3rd party services we rely on means that something is constantly acting up and there's not a lot we can do about it. Our API client code for each service we use typically contains _at least_ one service-specific hacky workaround to keep things running in the face of bad behavior.

The frontend team has no on-call rotation despite causing plenty of bugs on their own. Backend engineers are expected to triage what are clearly frontend problems. We stood up a lot of observability tooling for the frontend but it took years for them to even start to use it.

More than anything, it feels like the moment I stop championing the issue, everyone stops paying attention and the on-call experience reverts to the mean. Other on-call engineers just sort of stop boyscouting and let the chaos wash over them while focusing on sprint obligations (can't blame them), and leadership takes their eye off the ball to chase growth (also can't blame them). Hugely fucked lack of accountability and the buck eventually stops at whoever is the poor guy holding the pager that week.

Yes, this is probably the #1 reason. Alerts go off, but you don't even know if they are for real issues this time.

“Hi, welcome to being on-call, watch these channels and respond to any alerts that fire, except those alerts, you can ignore this subset of alerts, also you can ignore this alert unless it fires multiple times in quick succession, and this alert only matters if….”