A personal horror story: I have an on-call shift that is 24/7 for one week, only 3 other people are on the shift. Alerts are frequent, noisy, and happen almost every night so you are practically guaranteed to never sleep fully that week, and the sheer breadth of services + teams we’ve accumulated and lack of any clear specificity in alerts means that I’m almost always at least somewhat confused as to if something is actually broken and, if so, how I actually fix it, even after 3 years (two of which when I lived alone during the pandemic). This was my first job out of college too, I was fully convinced that if I fucked up even once or called the wrong person I would be fired and all of the effort I put into getting this career would be meaningless.
I didn’t even get any prep or mentorship, they just suddenly put me on-call during a major product launch. No extra pay or time off btw, just gotta continue work if you were up all night trying to fix some thing with vague priority & vague symptoms (too much latency on random offline service I’ve never heard of that turns out to be a dev experiment, latency being caused by a laggy database that you find out by finding some random message in splunk and regexing it out and into a graph).
I’m definitely a changed person after it, I don’t really… react as much anymore and flinch every time I hear a default iPhone text notification or ringtone. I don’t know how to fix it either—I don’t know if our team has enough people to spread the load out and I can’t think of any better way to keep track of failures in this labyrinth of services, and onboarding people to the point where they can actually take an additional shift is usually 8-12 months. Even experienced people still get ambushed by new services with zero documentation.
Pros though: I don’t really experience much stress or uncertainty anymore in hard situations, and I seem to be much better at problem solving! I’ve also managed to keep my prestigious job with life-changing pay, which feels much more personally fulfilling than coasting at Google (even if it’s for the wrong reasons).