Being on call sucks

yeah thats true, there must be some kind of middle ground between an ops team like that and the can't go to the bathroom without your phone oncall we have though.

I used to work for a small systems integrator. We used on-call mobile phones, the "hot potato" was carried by the on-call engineer. Any actual time worked outside of core-working hours was paid back in the form of time-off-in-lieu. The salary generously reflected the being on-call requirement.

The other factor was that a "call-out" was not completed until the root cause was fixed.

I believe the real reason for the compassionate arrangements was that the owners of the business were former engineers and were even available to escalate calls to them if you got stuck. Our personal phones had everybody else's personal numbers in the address book, but we were never permitted to give them out to clients. Clients only had access to the "hot potato" phone numbers, which also received the various paged alerts, etc.

> Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second.

And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep.

It's happened to me, I felt awfully guilty for not waking up after a gruelling work day and some pages early in the evening. I was tired and had been asleep for around 2 hours, didn't wake up and the escalation policy took it up to my manager... I didn't get reprimanded or had any bad consequence from it, still the guilt made me feel like a failure and increased my anxiety when I'm on-call.

Forced means, they didn't announce it as you could not do it. It's was a "be on-call" or you are out.

In a call I was explicitly told "every company does it like this, if that's not ok you might not be a right fit for this company".

So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong.

At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid when you're oncall. The amount varies depending on how important the service is and the expected response time but can easily be 5 figures a year. Oncall period varies but a week at a time varies with hopefully 8-12 people in rotation.

Too few people and people get burnt out. Even if nothing happens on an oncall shift, it's an annoyance and a restriction on what you can do. Too many and people tend to forget what to do. So with a sufficiently large team you may end up with some people in the rotation and some people not. That's why the compensation is importatnt.

Particularly large, important and mature services may enjoy SRE support. You can't throw a service over the fence and have SRE deal with it. It doesn't work that way. It typically needs to have been running for at least 6 months and SRE needs to be satisified it's sufficiently reliable, stable and monitored with a good runbook. SRE support is globally distributed and typically means 8 hour shifts during normal hours.

The owning team will often still be secondary support.

Also code has to be owned by somebody. This may be a team but when I was there (some years ago now so it may have changed) this also meant 2 actual people (not just team aliases) had to be owners. This is to avoid abandonware. This very much is a support and oncall issue.

Facebook OTOH is a dumpster fire when it comes to oncall.

Not getting paid to be oncall is (IMHO) one of the biggest mistakes. The mantra is "it's part of the job" but that responsibility is not shared equally. That's the point of compensation.

My experience at Google was that issues were relatively infrequent. What I saw at FB however was that oncall could often be the only thing you did for the week. Noisy alerts, alerts caused by issues in downstream systems that you could do nothing about or would get ignored by their oncall, a bunch of issues raised that some would just ignore until they expired (or closed just prior to going out of SLA as "could not reproduce"), etc. You may also be dealing with code that nobody owns (or, rather, nobody takes responsibility for) for features that are live.

Plus the incentive structure, at least on the product side, was to ship new features. Oncall was often treated as just extra work you have to do on top of whatever else you're doing.

Obviously I didn't see how every team did it so none of this is absolute but I did see a reasonably high number of samples.

It's also worth noting that not everything at FB is like this (eg the Web Foundation people were and I believe still are outstanding). Also, in high-visibility outage situations you have highly knowledge individuals who can and do get involved and know the right people to push.

The FB equivalent of SREs is Production Engineers ("PEs"). There are less of these and more services at FB are supported by the SWEs than at Google (IME).

I got the impression that FB processes and culture were forged when the company had less than 500 employees and they never really adjusted to the greater scale. There are a lot of things that work very well. Oncall just isn't one of them. Nor is code ownership.