Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being on call sucks

bobbiechen.com

Post reply on HN
Print view View original

201–210 of 282 posts

Re: Being on call sucks

#201

Post by BeetleB »

cube2222 wrote: post #6

On-call is just fine as long as: 1. There is a rotation. 2. There are few pages, preferably the median should be 0 per week. 3. Spurious / non-actionable alerts get fixed right away (with very high priority) 4. You're not up more than 1 week per 1-1.5 month. 5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime. As wit…

> There are few pages, preferably the median should be 0 per week.

In interviews, always ask them:

1. How often is someone on call (typical was one week every 6-8 weeks).

2. While on call, how often do you get paged after hours?

3. What do you do to reduce the number of pages?

I suspect I've been rejected for merely asking the second and third question, but that's good!

Re: Being on call sucks

#202

Post by aledalgrande »

scottlamb wrote: post #189
sam0x17 wrote: post #100

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and…

and document anything and everything and write amazing runbooks

Re: Being on call sucks

#203

Post by mancerayder »

It's possible to tolerate on-call if:

You have the ability to make changes necessary at the infrastructure engineering stage, not at the ops ad-hoc response stage, to prevent or end problems that cause call-outs.

There are those that that view "DevOps" not as an engineering culture but as "people who fix problems in production" and even "do releases Friday at 10pm" and developers (or "engineers") as "people who make changes, and go to the bar Friday at 6pm." Companies that do this sometimes call themselves places that "move quickly, break often." Places like this, you hope never to work for.

Re: Being on call sucks

#205

Post by probablypower »

mancerayder wrote: post #203

It's possible to tolerate on-call if: You have the ability to make changes necessary at the infrastructure engineering stage, not at the ops ad-hoc response stage, to prevent or end problems that cause call-outs. There are those that that view "DevOps" not as an engineering culture but as "people who fix problems in production" and even "do releases Friday at 10pm" and developers (or "engineers") as "people who make…

You are probably making a good point here, but you're doing so in such a roundabout way that it isn't clear what your point actually is.

In the given context, how would you describe a company you would hope to work for?

Re: Being on call sucks

#206

Post by SomeCallMeTim »

I worked for one company that enforced on-call for the entire team. For a reason that's out of scope for this comment, it didn't apply to me, though it should have by the normal rules.

I loved working for the company, but at the same time I would have totally despised the on-call system as implemented.

The main problem was that I was working on client apps. My focus was an Android app, to be specific. There was a completely separate web team that developed the backend; we not only didn't work on the backend, we were prohibited from working on the backend. I never even saw the backend code. We even asked to develop part of the backend we needed at one point, and they refused to let us, telling us that we didn't understand their security requirements and therefore couldn't contribute.

And from what I heard from people who were in the on-call rotation, every single 2AM call was from some badly designed alarm. Designed by their team.

Yes, every one of those alarms was fixed. Some may have been only "fixed" the first time, but I got the idea that each did eventually get adjusted to not be completely spurious.

But what really offended my sensibilities was that the backend team was pushing for the client team to be on-call to cover for what seemed to me to be profoundly poor alarm definition. They were eager to get others on board to cover the on-call rotation because being on-call was such a nightmare.

Another comment [1] points out that the median number of alerts in a week should have been zero. It wasn't close, from what I could tell. And the whole "make them eat their own dog food" approach of putting the people on-call who are actually responsible for writing the code was broken by the practice of including people only loosely associated with the code (as in, we used it) in suffering the consequences of writing the code (or designing the alarms) badly.

In general, if I've broken something that's affected a site or product, I'm happy to fix it, even if it's after hours, though I prefer it to be on a "best efforts" basis rather than a "drop everything and work on it now" basis. What I don't want to sign up for is being roused out of bed at 2AM to fix problems caused by someone who isn't even on my team, where I wouldn't have even seen the PR that caused the problem or any of the related code, and there's absolutely no way I could have prevented it.

/rant

[1] https://news.ycombinator.com/item?id=32163155

Re: Being on call sucks

#207

Post by mancerayder »

probablypower wrote: post #205
mancerayder wrote: post #203

It's possible to tolerate on-call if: You have the ability to make changes necessary at the infrastructure engineering stage, not at the ops ad-hoc response stage, to prevent or end problems that cause call-outs. There are those that that view "DevOps" not as an engineering culture but as "people who fix problems in production" and even "do releases Friday at 10pm" and developers (or "engineers") as "people who make…

You are probably making a good point here, but you're doing so in such a roundabout way that it isn't clear what your point actually is. In the given context, how would you describe a company you would hope to work for?

My point is that on-call is OK if the people that are on-call are empowered to either fully prevent, or permanently fix, conditions that lead to on-call events. I'm talking about software development and infrastructure design, which I thought is the topic here.

I've done this for many years. The worst case scenario is that the people that are getting woken up at night don't have any control over the root causes of the wake-ups.

If you own and created the environment, and it wakes you up at night, that's one thing. If you're just on-call and responsible for other people's messes, that sucks.

Hopefully that's more clear!

Re: Being on call sucks

#208

Post by ggeorgovassilis »

atemerev wrote: post #196
ggeorgovassilis wrote: post #95

Earlier quoted context omitted.

I'd like to learn more about that, it seems relevant to my situation. Some context (I'm not implying that this applies to you): we have an (well above average) capable engineer with a hero complex. In the most difficult of situations, he'll dive in and save the day (or night) where everyone else fails. He's also the one in the team who resisted standardisation and automation the most which, in a way, lead in the past…

Yes, this looks like my case. This is why I had to quit the corporate workforce and start working as a consultant — it provides me enough novelty and suits well to my strengths. I realize that in a regular corporate environment, I can be a liability instead.

Thank you. Do you have advice for me how to create a more welcoming and productive work environment for my colleague? Edit: especially with respect to the automation/standardisation approach we've taken?

Re: Being on call sucks

#209

Post by bornfreddy »

DoughnutHole wrote: post #141
sokoloff wrote: post #125

Earlier quoted context omitted.

What I find interesting is that if you give employees a choice of 100 units of currency per year as salary plus 1 unit of currency for each week on call with an expectation of 1 week on call every four weeks versus 115 units of currency for the year with the same on-call expectations, some employees will feel better about the first arrangement and some will feel better about the second arrangement. Some people want t…

I'd say the issue for people who are inclined to pick the first arrangement is one of trust. If you don't trust your employer not to abuse the second arrangement then you'd be a fool to accept it. With the first arrangement the employer is incentivised not to exceed the agreed upon expected number of on-call weeks because they are financially penalised for additional on-call time. With the second arrangement the empl…

I am firmly in the camp that would pick the first arrangement, but not because of (lack of) trust. It is partially due to psychological effect on me (get out of bed, you are compensated for this) and partially because the incentives of the employer are better aligned with mine. They should view 24/7 operation as something that costs money, so that they are more willing to invest in robust solutions.

Re: Being on call sucks

#210

Post by zippergz »

One of the main reasons I quit being a professional software developer was the expectation of being on call. My sleep and my free time with my family are more important than any job. No matter how much I like the normal work, and how much they pay me, I'm done working jobs that insert themselves into my home life and ESPECIALLY that wake me up in the middle of the night. EVER.
Post reply on HN