Live data from Hacker News

Being on call sucks

bobbiechen.com

101–110 of 282 posts

Re: Being on call sucks

#101

My last gig, on call worked well, I thought, for a few reasons: it was our services that we wrote, it was 1 wk out of 6 that you were on call, we heavily prioritized fixing unactionable alerts and automating fixes -- every alert had a runbook entry that described the non-automated fixes, and while on call your or sprint commitments were not counted, That last point was very nice as it meant you could work on whatever…

It is OK being on call on your servers and your software.

I think it is also why less and less people having stuff on premise and going for SaaS/Cloud solutions.

If someone wants to have my software on their servers and me not having any access better they have dedicated person for running it and I don't care even if they pay $1000 per hour - I am still not dealing with server I don't know talking on the phone with admin that has no clue how my software should be configured.

Re: Being on call sucks

#102
Story time:

This one time I was hired as a tier-3 Unix & Linux support engineer at managed-hosting provider who shall not be named. This was back in my prime, I'd already had 10 ~ 15 years industry experience, so I was hired to deal with the really nasty problems that percolated up the chain, and to solve those problems via ongoing process-improvement loop. Over time my job got easier and easier, because lower tier engineers gained training and better standard, procedures, etc... I was on a two person team, we traded the on-call duty every other week. One day the other person left the company, and that began a comedy of errors...

We tried to fill the position both internally and externally, but the position was mostly vacant. Meanwhile I was covering the on-call 24/7 on a tentative basis, until we hired a permanent replacement. Welp, long story short... I fell victim to my own success. Management observed that there was no apparent disruption to the tier-3 area of operation, and that was mostly true, so they decided to eliminate the redundancy. BIG MISTAKE!

I politely told management that I'd no longer be covering on-call 24/7, and go back to every other week; That I was burnt-out waiting for a replacement, and not being able to plan vacations, or even experience off-hours serenity for fear of the on-call phone ringing. That didn't go very well, I was told that I'd be subject to disciplinary action, and they would make accomodation for vacations, but that I had to remain on-call 24/7.

So I decided to become a party animal, every other week. As soon as I left work for the day, I'd start getting drunk. I had parties every night, or went to parties, bars, or whatever... on the weekend I'd go camping at state or national parks with 1-bar of signal for on-call device, I'd be on a boat in the middle of a lake an hour away from my laptop back at the cabin. Stuff like that, pretty much I'd make myself available 24/7, but there was zero assurance I'd be sober or whatever... every other week.

This one time I went into work, and somebody was joking to me about something in context, like as if I knew what they were talking about, but I had no idea. Apparently I was engaged the night before while blackout drunk, and was semi-belligerent with the tier-2 engineer, but was able to get whatever emergency escalated issue resolved in a most anti-climatic manner. The story I heard was something about slurring-out linux commands from a noisy bar, and threatening to drive home (drunk) to ssh into a customer's system. But whatever instruction I gave over mobile did the trick, and the company barely met the contractual SLA for that incident!

So my plan kinda backfired. Management figured out what was going on, and while being applauded for doing my job, I was formally written-up for being drunk on-call. So I responded by stating that's only every other week, and my on-call rotation is documented on the company calendar. I asked if I was never permitted to drink off-hours while employed at the company, and that got HR involved. Apparently it was not entirely legit to be on-call 24/7, nor to intrude into my personal life to such an extent. So I was asked to sign something, an amended employment contract. I refused, and suddenly my performance reviews tanked the next few quarters, and was eventually let-go via a round of "layoffs". The entire tier-3 was eliminated, and they went with a tier-0 ~ tier-2 hierarchy (whatever that entails). No harsh feelings, I'm still in touch with many peeps.

It was at this point I transitioned from supporting Linux to making Linux. Pursued my passion, and started working full time as open source developer, and lived happily ever after not being on-call ever again. Or so I thought. Turns out software development occasionally has grindy rushes to meet deadlines, the so-called "crunch" culture, and instead of having a company provided on-call device... work peeps were calling my private number, in the middle of the night.

And so it goes...

Re: Being on call sucks

#103
post #39

Earlier quoted context omitted.

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

(not the OP but my 100 Croatian Lipa) Depends heavily on how stable app/system is. I'm from PeopleSoft / ERP world. The same PeopleSoft HCM base application would generate one off-hours pager alert a month at one client; and half a dozen a night at another client; due to different customization/implementation/complexity of business logic and data. Any on-call/on-shift rotation system must be viewed through the lens o…

Another data point from a startup, we have a few people also globally distributed on on-call, we use each other as backups in case as the commenter suggested you're drunk at a bar or camping. Our site doesn't break much (haven't had an incident for over a month now, maybe something happens once every 1-3 months and it's usually not severe) and we have redundancy in the schedule so we've come to not feel so neurotic about it.

Re: Being on call sucks

#104

On-call is even worse for people with disabilities. I quite literally can't do it unless I stop taking my antipsychotic. Under ADA, I can not be placed on call, regardless of policy, nor can I be discriminated against for that. On-call is not an essential function of being a software developer, with very few exceptions—all of which have nothing to do with "policy" or "fairness". Needless to say, companies (and some c…

Not to detract from you comment, but you don't have to have a disability. Some people simply cannot do on-call, disability or not. I had a co-worker at a previous job, he did two or three on-call rotation and told our boss that he couldn't do it. Mentally it's simply to much for him, especially outside business hours where he felt alone with to much responsibility. In terms of abilities and qualifications he was abso…

> Not to detract from you comment, but you don't have to have a disability. Some people simply cannot do on-call, disability or not.

Imagine being a single parent; or needing to support elderly or disabled family.

Re: Being on call sucks

#105

On-call is even worse for people with disabilities. I quite literally can't do it unless I stop taking my antipsychotic. Under ADA, I can not be placed on call, regardless of policy, nor can I be discriminated against for that. On-call is not an essential function of being a software developer, with very few exceptions—all of which have nothing to do with "policy" or "fairness". Needless to say, companies (and some c…

Not to detract from you comment, but you don't have to have a disability. Some people simply cannot do on-call, disability or not. I had a co-worker at a previous job, he did two or three on-call rotation and told our boss that he couldn't do it. Mentally it's simply to much for him, especially outside business hours where he felt alone with to much responsibility. In terms of abilities and qualifications he was abso…

[deleted]

Re: Being on call sucks

#106
post #6

On-call is just fine as long as: 1. There is a rotation. 2. There are few pages, preferably the median should be 0 per week. 3. Spurious / non-actionable alerts get fixed right away (with very high priority) 4. You're not up more than 1 week per 1-1.5 month. 5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime. As wit…

> 4. You're not up more than 1 week per 1-1.5 month.

That seems excessive if you're expected to be able to log into your work system within X minutes.

Having to be essentially home, near a computer, 25% of the time (1 week out of 4) is a pretty heavy burden, especially for people who prefer to be out, rather than home.

Re: Being on call sucks

#108
Compared to the article, many on-call situations could suck even more.

In a past assignment, I was on-call for a service managing thousands of customer instances each with a different version of an highly customizable application (basically an SDK) developed by another team.

The result was on-call being basically a working day, getting paged on average every hour and at worst needing to track 2 or 3 war rooms in parallel.

Also you felt powerless because while being accountable for the availability of the service, a lot of the issues actually came from customer implementations overloading their instance or product bugs which would get ignored for years and if a fix was done it would take even more years to be deployed due to the difficulties in updating the application.

The only saving graces were:

* It was in a large international org, so it was "follow the Sun", no midnight calls, only 8 hours shifts (11:00 to 19:00)

* I live in a country were it's a legal requirement to compensate employees for being on-call

Now, I've switched to a team were we handle on-call on services we control end to end (code and deployment), and it has been far less stressful.

Re: Being on call sucks

#109
post #104

Earlier quoted context omitted.

Not to detract from you comment, but you don't have to have a disability. Some people simply cannot do on-call, disability or not. I had a co-worker at a previous job, he did two or three on-call rotation and told our boss that he couldn't do it. Mentally it's simply to much for him, especially outside business hours where he felt alone with to much responsibility. In terms of abilities and qualifications he was abso…

> Not to detract from you comment, but you don't have to have a disability. Some people simply cannot do on-call, disability or not. Imagine being a single parent; or needing to support elderly or disabled family.

I didn’t even think about that, but absolutely. I’m not a single parent, but my wife works in retail and just those late hours make daycare pick ups, late afternoon meeting, on-call and incidents a major hassle.

Re: Being on call sucks

#110
I've didn't got many but the few "events" that pulled me to help, one had the aim to almost screw up great a romantic date friday night, and the other had even more accuracy (details NSFW).

Being on-call is a deal breaker / non-starter for me.

I see myself helping the guys in operations to stay happy and well equiped to deal with keeping the service up but I am not doing operations myself.

Ever.

A challenge for this is that when the devs keep being hammered with new features and they cannot allocate to improve the system defensivness against these "events" then the thing gets tricky.

Post reply on HN