Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking Up with On-Call

reflector.dev

Post reply on HN
Print view View original

181–190 of 202 posts

Re: Breaking Up with On-Call

#181

Post by AdieuToLogic »

jjice wrote: post #91
AdieuToLogic wrote: post #26

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system.…

The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center". It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years…

> The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center".

That is a great philosophy to have when supporting a prod system off-hours, one which I fully subscribe.

> While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big[sic] different than the one you've experienced (which may be the more common one, I'm not sure).

My underlying thesis is not with being on-call, but instead expecting a developer to perform their non-support duties in addition to being on-call. The worst case scenario of this is when the on-call developer is also tier one support.

If an organization wants to have developers perform SRE duties, presumably due to deeper understanding of a system, fine. Assign them to support and suspend development responsibilities during same.

Just my $0.02

Re: Breaking Up with On-Call

#182

Post by AdieuToLogic »

procaryote wrote: post #63
AdieuToLogic wrote: post #26

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system.…

If the code is important enough that it needs to be fixed if it breaks at 03:00, someone needs to be on-call If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking. I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team an…

> I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call.

This is a leadership problem, not a development problem. If stakeholders prioritize resolving production issues and development teams do not address the objectives set forth, then there are bigger problems afoot.

> Pain from a problem needs to live where the problem can be fixed.

Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

0 - https://nielsbohrmann.com/prices-law/

Re: Breaking Up with On-Call

#183

Post by AdieuToLogic »

jyounker wrote: post #158
AdieuToLogic wrote: post #26

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system.…

Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation. The organization needs to incur financial penalties when the on-call staff have to respond at three AM. This aligns the organizations motivations with the operational motivations. When managers see an on-call-incident line-item going up then they're more willing to…

> Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation.

As I mentioned in a peer response:

Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

0 - https://nielsbohrmann.com/prices-law/

Re: Breaking Up with On-Call

#184

Post by dharmab »

dkdbejwi383 wrote: post #106
spiffytech wrote: post #83

Earlier quoted context omitted.

Any particular reason you can't handle incidents while out and about? I know it varies by situation. When I've been on call I've been able to mostly go about my life. I just had to keep my laptop close, stay in cell signal, and accept I would sometimes have interruptions (typically brief). We fought to keep them infrequent enough that they didn't ruin our lives.

I do long(ish) distance running as a hobby - it's not feasible to take a laptop out on a two hour run. If I want to go meet a friend for a drink or food, I have to lug around a backpack, keep an eye on it to make sure it's not stolen. If I wanted to have a beer or wine, I can't because I may need to work at any point. Favourite band is performing? I suppose you could take a backpack and the laptop to the venue, but a…

> If I want to go meet a friend for a drink or food, I have to lug around a backpack, keep an eye on it to make sure it's not stolen. If I wanted to have a beer or wine, I can't because I may need to work at any point.

If this is a stated requirement from your employer, talk to a lawyer. This is a common litmus test for whether you need to be paid while on call, even if you aren't actively working. Depending on the jurisdiction you may be entitled to pay (or trigger a relaxation of your company's policies).

Re: Breaking Up with On-Call

#185

Post by brigandish »

Xylakant wrote: post #40
brigandish wrote: post #23

Earlier quoted context omitted.

Who, specifically, would that picture offend or change protect?

People that have been interned in this or one of the other camps, or their descendants. It’s one generation ago, people born and raised in camps are still alive. George Takeo for example was born in one of the segregation camps. It’s a low stakes change. Nobody assumes harmful intentions from the author - I would not have recognized the picture either. But now that it’s been pointed out that it’s from a site where pe…

You have conflated two arguments:

1. That it's harmful

2. That it's easy to do something else

(1) is the important point, (2) is irrelevant. On the important point, you seem to have assumed several things:

a) that it is harmful

b) that anyone to whom the assumed harm would occur would see it

c) that they would know what the camp looked like

(a) I see no evidence for this, in fact, research suggests the opposite[1]

> The strange paradox about triggers and PTSD – and this is true for all anxiety-related disorders – is that avoiding triggers makes the disorder worse, not better (Jones et al., 2020). Being exposed to small instances of one’s triggers, in a safe environment such as therapy where one can be helped to process the situation, is a way to gradually become less reactive to those triggers (APA, 2013). Many people have learned to reduce their reactivity to psychological triggers through this process, called exposure therapy.

and (b) really is a stretch, are we to believe there are people in the tech industry who are on-call and were interned in WW2 and will read this article and will have some kind of meltdown because of it? Why would their descendants react badly to it? Unless they have a pre-existing condition, that's untenable, and if they have a pre-existing condition then they should seek help for it.

And to (c), how would they even know what the camp looked like if they're so triggered by the thought of it? They have an aversion to it.

No, none of that makes sense, and cloaking it in "compassion" or trying to handwave scrutiny away because it would be a "low stakes" behavioural change won't hide it.

> Apart from that, I cannot associate with the picture either

We are intelligent people, the link is really simple to make. There being "better" choices does not make the picture a non-sequitur.

I'm sure I'm not alone in wishing this West coast American style of pop psychology being misapplied to real life would die a death.

By the way, George Takei (I believe that's who you meant) is 87, and he's not really in tech.

[1] https://www.berkeleywellbeing.com/triggers.html

Re: Breaking Up with On-Call

#186

Post by wesselbindt »

brigandish wrote: post #185
Xylakant wrote: post #40

Earlier quoted context omitted.

People that have been interned in this or one of the other camps, or their descendants. It’s one generation ago, people born and raised in camps are still alive. George Takeo for example was born in one of the segregation camps. It’s a low stakes change. Nobody assumes harmful intentions from the author - I would not have recognized the picture either. But now that it’s been pointed out that it’s from a site where pe…

You have conflated two arguments: 1. That it's harmful 2. That it's easy to do something else (1) is the important point, (2) is irrelevant. On the important point, you seem to have assumed several things: a) that it is harmful b) that anyone to whom the assumed harm would occur would see it c) that they would know what the camp looked like (a) I see no evidence for this, in fact, research suggests the opposite[1] >…

Wow, it's really important to you to have a picture of a concentration camp guard tower up.

Re: Breaking Up with On-Call

#187

Post by asdf6969 »

jjice wrote: post #91
AdieuToLogic wrote: post #26

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system.…

The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center". It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years…

In my experience we patch the hole in the tire and then get directed to patch a hole in another tire. Manager says work on the highest priority tire so now there are 2 partially broken tires but the problem is technically fixed. Repeat for 10 years and now it’s time to rebuild it “correctly”, but actually we just have two cars to patch since nobody knows what awful hacks we need to port over for “expected” behaviour

Re: Breaking Up with On-Call

#189

Post by apeescape »

stackskipton wrote: post #115
apeescape wrote: post #72

Earlier quoted context omitted.

Well, did those startups succeed? If not, I think it means the author was spot on in saying startups can't _afford_ that.

One did, two did not. None of due to engineering. Engineering rarely matters unfortunately but ability to weave a story and sell that story.

Fair point about the importance of storytelling, but I wouldn't say "engineering rarely matters" in the tech industry. Compelling vision can mask those problems for a long while, but at some point the wheels will fall off if engineers can't deliver.

Re: Breaking Up with On-Call

#190

Post by wvh »

dakiol wrote: post #62

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is…

This. And alerts are often just a fluke anyway. Sure many of us can step up and pull an all-nighter if it saves some company and you make some minor sacrifice to do something heroic. Being woken up several times at night for no reason but some metric that is a bit off is pretty soul crushing, and then come the knock-on effects in professional and personal life that you're always tired and demotivated during the day.
Post reply on HN