181–190 of 282 posts
So management does NOT bear the burden of rushed and unstable code.
Then management will only care about new features and not about stability. This will result in developers burning out over time.
I've been at my first software engineering gig for 4 years now, we don't have on call, and I've sworn to myself that I will absolutely never take a position with on-call. I'm a bit worried that it will hamper my career prospects, especially as I've moved to doing more backend work. But I just can't imagine being tied to a work phone on my personal time -- I have a hard enough time enforcing work/life balance as it is…
So honestly, it probably won't, depending on how far you want to go.
But lots of the broken teams people are describing here are easily fixed by mandating that the tech lead / architect / whatever participate equally (or more-than-equally) in the on-call process. It's the simplest way to get people with leverage to push back on POs / executives who see it as just a cost to balance.
So if you want to have one of those positions one day, and you want to do it well - you should want to do some on-call, because you need to know if the systems and processes you're designing / managing are hell for those on-call.
Earlier quoted context omitted.
The problem with informal "people will notice" reward schemes, even when administered well (and don't take that for granted, it's easy for them to become popularity contests!), is that they encourage bad work-life balance among more junior employees. If you regularly have people popping up at 10 PM to fix things, and you don't have any formal recognition of the fact that they've gone above and beyond, new hires who w…
Going above and beyond what your fellow humans are willing to do is a pretty tried-and-true way to get ahead, whether you’re working for yourself or as an employee.
Earlier quoted context omitted.
You're using "never" everywhere here. That is in my opinion the main red flag here. On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds ha…
IMHO the whole notion of "week of on-call" is ridiculous. Being on-call ready is a shift of (hopefully low-intensity) work time. You can do 8 hour shifts, 12 hour shifts, 24 hour shifts, but you can't do 168 hour shifts for monitoring something, anything - that's impractical and should be illegal. There are (or should be) some minimum standards of rest time that every human being must get, and being on-call is not it…
Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…
That's only half the story. You should also compensate employees for being on call. As the article notes, it is a significant chore.
Unpaid on-call time is exploitative and illegal in most jurisdictions.
So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong. At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid wh…
> alerts caused by issues in downstream systems that you could do nothing about How does Google deal with issues caused by downstream systems causing alerts?
So if you're having an issue with Spanner, first it's likely not a bug in spanner. If it's an outage, somebody has probably already been paged. But if not, paging someone responsible will be answered quickly and treated seriously.
You could've unexpectedly gone over quota on something. More often than not you can alleviate that with temporary quota while you resolve your issue (by reducing your usage, getting more permanent quota or both).
A big part of this is that it's a cultural thing.
Earlier quoted context omitted.
What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. There should be no incentive to require or avoid "free" work, only advance planning of who'll be on call (with more available engineers doing more turns).
> What do you mean by "expectation"? One day on call, one day paid overtime; anything else is abusive. I don't think being available for work if an unlikely emergency comes up usually justifies the same comp as actively working that entire time.
Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of…
* fix the alerts so they reliably page when there's a SLO-worthy problem and only then.
* test the restore system so it works smoothly when needed at the necessary scale.
* add safety checks to prevent the need to use those backups in the first place.
* get to the root cause of yesterday's outage and prioritize the 9–5 engineering work to ensure it won't happen again.
It's awful to work for a place where heroics are often necessary and unrewarded. I still don't like working in a place where heroics are often necessary, even if they're celebrated. They're often avoidable.
So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong. At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid wh…
Note that the on-call bonus is not entirely known. I had managers try to put my team "on-call" for a product and after I explained to them how Google actually did it, they suddenly said "oh, it's not really on-call. You just have to be ready to answer the pager at any time and respond". I was also on what was one of the most dysfunctional on-calls at the company- keeping several distributed clusters of unique busines…
No manager should give you any resistance to giving away free money from the company to their team. If they ever do you know where their loyalties lie: with their management chain and producing the appearance of efficacy.
A manager should be fighting to give the team any on-call pay they're due.
Related side story: annual bonuses were (and maybe still are) calculated based on salary, level (ie target percentage) and ratings. It quickly becomes known what the base rate is so you can calculate everything. After this, your manager has a pool of extra money and a bunch of sliders for their reports. From that pool of extra money, they can distribute it evenly, weight it towards particular people, etc.
They can even take money away in this process from some people to give it to other people.
But because the formulae are all straightforward, this should be obvious. I have seen:
- Managers take away money from some people to give it to their favorites;
- Give all the extra money to one person; and
- (This is the crazy one) Not give all the money away. That is to say they'd rather not give away this free money and return it to the company. I've literally seen this happen.
That's why I mention it: any sign of a manager not giving their team everything they can should be a massive red flag.