Live data from Hacker News

Breaking Up with On-Call

reflector.dev

151–160 of 202 posts

Re: Breaking Up with On-Call

#151

Earlier quoted context omitted.

My on call experience required that I had to be able to respond within 10 ten minutes of the call, with 24/7/365 coverage. But if I couldn't get the issue resolved remotely it meant that I'd have to be in the office lab to recreate and reproduce the problem. It effectively restricted my movements personal movements to stay within commute distance of my office, and that includes all my vacation time as well. That was…

I don't understand why on-call is normal. It's a huge mental burden on employees, which is especially an important issue in modern times of widespread mental issues, just so that some shitty mobile app can be available 24/7/365. If your business is important enough to have on-call, then you should have dedicated employees covering night shifts and nothing else, effectively limiting it to someone's office hours, effec…

> If your business is important enough to have on-call, then you should have dedicated employees covering night shifts and nothing else, effectively limiting it to someone's office hours, effectively removing on-call.

Easy to say when you envision it as someone's office hours and not your office hours. If my employer gave me a choice between a normal shift plus oncall or a night shift with no oncall, I'd pick the on-call in a heartbeat.

Re: Breaking Up with On-Call

#152

Just hire an SRE. If you want to piss off your team and incur more costs than the annual salary of one decent SRE... ask your team to do on-call rotations. In this day and age if you're a properly capitalized startup you should have the budget for one decent SRE hire. They often times double as a technical writer and are great at preventing knowledge silos in teams / members of a small team since they have to sort of…

I don't follow how a single SRE hire would be able to replace an entire team's oncall rotation.

Re: Breaking Up with On-Call

#153

I have no idea what article was trying to convey since on call was poorly defined I also couldn’t take it seriously when article opened with this. > Startups cannot afford engineers to baby sit software, big tech does. Say what? As ops person, I’ve seen multiple startups where devs are drowning in operational issues because software was written enough for feature MVP, ship with poor testing then constantly poke at wi…

> I have no idea what article was trying to convey

An ad for their consultancy. It's a call to action, not a balanced analysis.

Re: Breaking Up with On-Call

#154
Typical HN blogspam. A blog post that makes vague but incredibly certain claims, generalizing about an entire industry, with no evidence outside a single person's limited experience. Yet there he is at the end of his post, asking people to hire him for consulting. But it's perfect for HN, because it gives people in the comments something to complain about, driving up engagement.

I will admit, though, the SaaS tech industry as a whole has a problem. The cargo cult of tech has convinced itself that SaaS is special. That it's never done. That it must be never finished, never stable, always changing, always breaking. Like it's some immutable law of nature.

This of course is a convenient lie. Before every business was an online service, software worked like every other product in the world. You built it to do a thing, you tested it thoroughly to ensure it didn't have defects, and then you coordinated one large release. No constant changes or late Friday deploys. No schema changes destroying columns. There was no on-call, because there was no 24/7 service. There were floppy disks and CDROMs, and people running software on their own computers, and a very small number of 24/7 enterprise systems shared by lots of users (mostly run by ISPs, and one or two large tech companies and industry bodies).

But you can deliver online software like it's desktop software. You can ensure your software is bug-free before you burn your master disk. But then you don't get to skip all those time-consuming tests. Then you don't get to ship your features faster than your competitor. Then you don't get to do A/B testing and micro-tweaks and experiments and feature flags and blue/green deploys. We want to do all the things, at any time, with no consequences. So on-call is still a thing, because people want to have their cake and eat it too.

The actual infrastructure on the backend shared by millions of users? Not actually hard to maintain. It's the same shit as 20 years ago, but much, much easier. Make sure the disks don't fill up. Auto-scale the VMs and containers. Design your apps to not exceed network bandwidth. There's really not much else to break. The only thing that breaks a system is changing it, or bugs from not enough testing. So do the testing, and the bare minimum of autoscaling in the cloud, and nothing should ever break in production.

But constantly mutating SaaS, without due diligence, is too addictive. Nobody's going to abandon the freedom of doing whatever the fuck they want, just to keep from waking people up in the middle of the night. The business people sure as fuck aren't going to abandon their "competitive advantage". And devs who don't care if their software works or not don't want to abandon untested deploys from their laptops. So on-call is here to stay.

Re: Breaking Up with On-Call

#155
post #64

Earlier quoted context omitted.

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT 's job, whereas I'm SE . No thanks

Aren't hackers supposed to be a curious bunch? Is that really the only way you can imagine unions working? Can you not see the imbalance of power between a single individual and the corporation that employs them? Unions are fundamentally about balancing that power dynamic.

Re: Breaking Up with On-Call

#156
post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

I once had a job with a lot of 2-4am wakeup outage calls. The timing was perfect such that you can't fall back asleep generally.

An aggravatingly large percent of them could be resolved by voice over the phone by walking the offshore support team through the same 2-3 runbook items.

"Did you look at the log... I see, OK are you looking at it now? Does it say X? Did you do Y? Good now? Great, goodnight."

"Did you try restarting it.. ok then try that now. Is it good now that you restarted it? Great, I'm going back to sleep"

Ironically we'd have less of these outage calls when the offshore person went on holiday because they'd send one of the competent NY support staff over for 2 weeks. Slept like a log every time.

Re: Breaking Up with On-Call

#157
post #80

Earlier quoted context omitted.

I don’t see how this changes the problem where there is an expected guarantee of a rapid response except that now two people are expected to be available and would now need to directly coordinate in order to ensure one person’s going for a swim doesn’t interfere with the other’s WoW raid.

That's more or less what my team does. It works well. At least much better than saying you can't for for a swim at all.

I guess to me that seems worse because that’d effectively double the number of off-hours accountability per teammate. Not only do you need to be first on call for your primary hours, therefore severely restricting the quality of your “free time” but now you ALSO have to be secondary on call for that irresponsible coworker that goes afk without properly communicating for 2 hours, dipping twice into your actual free time.

Re: Breaking Up with On-Call

#158

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system.…

Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation.

The organization needs to incur financial penalties when the on-call staff have to respond at three AM. This aligns the organizations motivations with the operational motivations.

When managers see an on-call-incident line-item going up then they're more willing to tell product to take a hike.

Re: Breaking Up with On-Call

#159

Earlier quoted context omitted.

So my coworker who was a UAW member who told me stories about sleeping on the roof, and being reprimanded for moving a desk to retrieve a pen...was trying to dupe me?

Why are you getting all of your thinking from one co-worker?

So I should ignore evidence directly from the source of one of the largest unions in the US, because it doesn't support your view? I should only accept evidence from your trusted sources? Ok.

Edit: or my UPS friend who told me how the union box loaders would falsely claim alcoholism or drug addiction before being fired so they could abuse the union "protection" that was given to them? Is he trying to dupe me too?

Re: Breaking Up with On-Call

#160
The author's description of how on-call works at "big tech" seems to be a complaint about a specific tech company, and the author is hastily generalizing to the rest of the industry.

His experience has nothing in common with my on-call experience working at a different "big tech" company.

Post reply on HN