Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being on call sucks

bobbiechen.com

Post reply on HN
Print view View original

131–140 of 282 posts

Re: Being on call sucks

#131

Post by hrudham »

bpodgursky wrote: post #116
sam0x17 wrote: post #100

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

Unless you're in a really dysfunctional startup, the employees who voluntarily jump in after-hours to fix OC issues are going to be very quickly compensated with raises and refresher option grants. Far better payout generally than a few bucks of overtime. It becomes really obvious who is holding the company together and who is coasting, relying on the "senior" (ie, anyone who puts in the effort) engineers to keep the…

I respectfully disagree. I did exactly this for a number of years. A colleague (and is now friend since we've both left said company) made it clear he would not work overtime without compensation (and never did), but his day-to-day work was excellent. He knew he had a skill they required, and was good at it (and our skill sets were the same).

Our careers both grew at about the same pace for four years. He became known as the go-to guy for green-field projects, while I was the guy you could put onto over-budget tight-deadline projects to rescue them. I worked a lot of weekends for my troubles, and I always envied how he ended up on the "fun" projects. I learnt a lot from him. As a professional, some self respect is required, or you will be abused.

Re: Being on call sucks

#132

Post by SpicyLemonZest »

throwk8s wrote: post #106
cube2222 wrote: post #6

On-call is just fine as long as: 1. There is a rotation. 2. There are few pages, preferably the median should be 0 per week. 3. Spurious / non-actionable alerts get fixed right away (with very high priority) 4. You're not up more than 1 week per 1-1.5 month. 5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime. As wit…

> 4. You're not up more than 1 week per 1-1.5 month. That seems excessive if you're expected to be able to log into your work system within X minutes. Having to be essentially home, near a computer, 25% of the time (1 week out of 4) is a pretty heavy burden, especially for people who prefer to be out, rather than home.

It's a heavy burden, and a lot of teams might want to consider a longer interval, but there's a lot of legitimate scenarios where it's just not feasible to distribute an oncall rotation among 8+ people. I would definitely point more towards 1 in 6 as the ideal minimum.

Re: Being on call sucks

#133

Post by 4ggr0 »

jddddd wrote: post #123
dml2135 wrote: post #114

I've been at my first software engineering gig for 4 years now, we don't have on call, and I've sworn to myself that I will absolutely never take a position with on-call. I'm a bit worried that it will hamper my career prospects, especially as I've moved to doing more backend work. But I just can't imagine being tied to a work phone on my personal time -- I have a hard enough time enforcing work/life balance as it is…

It won't. On call is a symptom of poorly run company. It's a great signal that you should run far away from any place that requires it. Most software isn't as critical as we think, and the software that is, is expensive enough to have a properly sized staff. On call exists for the same reason game devs are paid shit and open source exists... Software engineers don't value themselves properly and love giving away free…

This sounds like a very software-engineery opinion. Not saying that you're wrong, I understand that on-call is not that important in software development. But as a System Administrator I think that on-call is useful, else we wouldn't notice any outages happening during the night and would then only start working on it in the morning.

Plus, we have customers who actually work during the night, be it timezones or specific industries, so we can't just ignore outages outside of our engineers work-times. And if you're selling SLAs to your customers with promised up-times etc. you better be able to detect and fix something ASAP.

Overall I think "On call is a symptom of poorly run company. It's a great signal that you should run far away from any place that requires it." is a bit too harsh of a statement.

Re: Being on call sucks

#134

Post by donarb »

Years ago, I was a second-shift operator in a computer center for an insurance company. We ran production jobs on an IBM mainframe. When jobs would crash we would write up the error on an ABEND form (IBM called crashes ABENDs for ABnormal END), collect the printout and call the programmer responsible. One night a production job crashed late in my shift about 10:30-11 pm and I woke up the programmer responsible. He seemed really groggy and it took a few minutes for me to describe the crash to him. I would always try to be helpful and suggest options for recovery (you got to know the programmers and what their recommendations would be based on the type of production job). Usually they would hold the job, restart it or say they were coming in to fix, this was back in the day where if they could log in remotely, it was with a clunky CRT terminal.

The programmer told me to just restart the job, I noted that on the form. I came in the next day and my boss called me into his office, his boss was there too. They wanted to know why I restarted the job, which caused all kinds of corruption to the database. They had spent the better part of the day recovering the database, then running the batch job, which meant that that system was unavailable for use by the agents.

The programmer swore up and down he did not tell me to restart the job, said I never called him! He was that deep into sleep. But on the form I noted the time I called him and his response to restart the job, so they believed me.

This highlights a problem with people responsible for multi-million dollar systems being woken up in the middle of the night and having to make quick critical decisions.

Re: Being on call sucks

#135

Post by ford »

I think more teams should recognize that some features/products/functionality (especially at large companies is not worth paging for.

Obviously if a customer can't access their bank account or sell a stock then someone needs to be woken up, but if an endpoint has high latency or a business can't fire off a marketing campaign ASAP it can mean entire teams of engineers can avoid the worst parts of oncall - the feeling of dread of waiting on the "ba-dum" in the middle of the night and questions about making plans outside of work - just by shifting oncall to noisily page during business hours only.

Re: Being on call sucks

#136

Post by PeterisP »

cube2222 wrote: post #38
nickjj wrote: post #19

Earlier quoted context omitted.

Finally, somewhere that I fully agree with. Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available. Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work…

You're using "never" everywhere here. That is in my opinion the main red flag here. On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds ha…

IMHO the whole notion of "week of on-call" is ridiculous. Being on-call ready is a shift of (hopefully low-intensity) work time. You can do 8 hour shifts, 12 hour shifts, 24 hour shifts, but you can't do 168 hour shifts for monitoring something, anything - that's impractical and should be illegal.

There are (or should be) some minimum standards of rest time that every human being must get, and being on-call is not it, every employee must get an opportunity to fully disconnect for non-trivial time during every single week. You do your shift, and then get at least 12 hours off (with your phone off) before being available for work or work calls again.

Re: Being on call sucks

#137

Post by bobkazamakis »

angarg12 wrote: post #129

This article should be called "being on a sucky team sucks". It sounds like OP just has experience with one bad on call. I've also been there. I even heard of teams with +100 high severity issues per week. But it doesn't need to be. My current team has the best on call I've ever experienced. It's a mix of a lucky product and some discipline. It isn't rocket science really. If a team is drowning in ops here are two ea…

>1. Whoever is on call, their job isn't only to answer pages, but also to improve the system

Way to make being on-call more punishing as well as detract from both goals -- your clients receive worse support and the amount of work done on the fix is inversely proportional to the time you're spending with clients who are calling you because there is a problem, not because they just want to talk.

2 actually works; tech debt hackathons are a great thing, and sometimes ops and dev hands don't communicate as effectively as they need to throughout the SDLC.

Re: Being on call sucks

#138

Post by Aeolun »

> Does anyone have One Weird Trick™ to fix it?

My one weird trick is to have a zero tolerance policy for flaky monitors/tests. If it’s not accurate, we either have to drop everything else and fix it, or disable that alarm entirely.

Like they say, normalization of deviance is real, and the only way to fight against it is to have every form of deviance be a problem.

Re: Being on call sucks

#139

Post by [deleted] »

Broken_Hippo wrote: post #35
cube2222 wrote: post #8

Earlier quoted context omitted.

If it's in the contract you sign, then it's already priced in. I don't see much value in specifically outlining which part is base and which is for oncall, if oncall is mandatory.

Because if you are supposed to be prepared for sudden work outside your normal work hours, you should be compensated outside of your standard pay. Period. And sure, it is priced in because the companies can get away with it. This doesn't mean it is right or fair as the price is always going to be in the company's favor and rarely fair to the employee. For example: My brother worked for a US railroad. He didn't have a…

[deleted]

Re: Being on call sucks

#140

Post by sakopov »

masukomi wrote: post #130

despite working on a financial product where a production fire could actually cost normal people money (worst case scenario) I don't mind being on call at all. Why? Because I never get called. Because we wrote tests. Because we were effing careful. Because we have safeguards in place. Because IT has put hardware redundancies in place and because we have circuit breakers. Being on call only sucks if you're being made…

Absolutely this. I've worked on teams that do on-call rotations and teams that don't. What I typically find is that when you're on-call you work hard for your time off-work to be peaceful. Everything becomes infinitely better - code reviews, testing, deployments. When you do have an on-call issue that is beyond something simple and transient, it gets brought up in incident reviews for all engineering teams. This keeps everyone accountable and focuses team's efforts on patching anything that causes on-call pain. You really don't want to have to say anything in this meeting. And if you do on-call right and focus on quality, you're basically never doing anything during your rotation. On the flip side, when a team doesn't do on-call rotations the code quality suffers greatly.
Post reply on HN