121–130 of 282 posts
If it wakes me, I should have built it better. So I work really hard to build things that don’t page. So I actually want to be on call for the quiet and reliable things I built, in order to experience the reward for all that effort.
Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…
Unless you're in a really dysfunctional startup, the employees who voluntarily jump in after-hours to fix OC issues are going to be very quickly compensated with raises and refresher option grants. Far better payout generally than a few bucks of overtime. It becomes really obvious who is holding the company together and who is coasting, relying on the "senior" (ie, anyone who puts in the effort) engineers to keep the…
I've been at my first software engineering gig for 4 years now, we don't have on call, and I've sworn to myself that I will absolutely never take a position with on-call. I'm a bit worried that it will hamper my career prospects, especially as I've moved to doing more backend work. But I just can't imagine being tied to a work phone on my personal time -- I have a hard enough time enforcing work/life balance as it is…
On call is a symptom of poorly run company. It's a great signal that you should run far away from any place that requires it.
Most software isn't as critical as we think, and the software that is, is expensive enough to have a properly sized staff.
On call exists for the same reason game devs are paid shit and open source exists... Software engineers don't value themselves properly and love giving away free labor.
Earlier quoted context omitted.
I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not…
Quoted post unavailable.
Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…
Some people want to see everything broken out and take a very transactional approach to work; others prefer the simpler approach but with less clear linkage to a specific piece of work.
Earlier quoted context omitted.
Unless you're in a really dysfunctional startup, the employees who voluntarily jump in after-hours to fix OC issues are going to be very quickly compensated with raises and refresher option grants. Far better payout generally than a few bucks of overtime. It becomes really obvious who is holding the company together and who is coasting, relying on the "senior" (ie, anyone who puts in the effort) engineers to keep the…
The problem with informal "people will notice" reward schemes, even when administered well (and don't take that for granted, it's easy for them to become popularity contests!), is that they encourage bad work-life balance among more junior employees. If you regularly have people popping up at 10 PM to fix things, and you don't have any formal recognition of the fact that they've gone above and beyond, new hires who w…
My last gig, on call worked well, I thought, for a few reasons: it was our services that we wrote, it was 1 wk out of 6 that you were on call, we heavily prioritized fixing unactionable alerts and automating fixes -- every alert had a runbook entry that described the non-automated fixes, and while on call your or sprint commitments were not counted, That last point was very nice as it meant you could work on whatever…
It is OK being on call on your servers and your software. I think it is also why less and less people having stuff on premise and going for SaaS/Cloud solutions. If someone wants to have my software on their servers and me not having any access better they have dedicated person for running it and I don't care even if they pay $1000 per hour - I am still not dealing with server I don't know talking on the phone with a…
Earlier quoted context omitted.
I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not…
Quoted post unavailable.
It sounds like OP just has experience with one bad on call. I've also been there. I even heard of teams with +100 high severity issues per week. But it doesn't need to be.
My current team has the best on call I've ever experienced. It's a mix of a lucky product and some discipline. It isn't rocket science really.
If a team is drowning in ops here are two easy techniques I've seen work well.
1. Whoever is on call, their job isn't only to answer pages, but also to improve the system. This solves the dichotomy of feature work vs ops fix. The ops fix is your job for that week.
2. Have team wide (even org wide!) fix-it days, where everyone works exclusively in operational issues.
Again, we didn't invent this. Look at Google's SRE books on reducing toil. You can adjust the ratios of feature vs ops work as needed.
And if you are in leadership, please acknowledge, celebrate and reward operational work. People tend to work in what they perceive as being valued.
Why? Because I never get called. Because we wrote tests. Because we were effing careful. Because we have safeguards in place. Because IT has put hardware redundancies in place and because we have circuit breakers.
Being on call only sucks if you're being made responsible for crappy software that breaks regularly.
That's a problem if you aren't empowered to make it better. If your team isn't encouraged to care. Better... find a place that does care. Find a place where people hate the idea of being awoken at 4 AM and do everything in their power to make sure it can't happen to anyone.
The last time i got called it was because AWS went down, and we couldn't do crap about that.