Developer on Call

I'm agree that real blame with real consequences will encourage people to clam up/hide evidence/start framing, but I don't think it can be 100% true/the only outcome because if this was the case then no consequences-based governance system would ever work. Also I admit it's a bit naive to say but maybe we should also be focusing on not hiring people that do egregious things when consequences arrive? A certain amount is normal but if you're working with people that destroy evidence and frame others when shit hits the fan... That's kind of a red flag no? No one hires for integrity anymore?

I also disagree about it wrecking trust -- a well built & fairly applied system should build trust -- it's when people put their trust in systems with no power/hidden manipulation that trust gets wrecked the fastest. You could even make it opt in, and tie bonuses to the risk taken by those who decide to have raises/promotions manipulated in the context of the system.

IMO this is basically just a sub-problem of the general "how do we govern societies" general problem and "don't have consequences" doesn't seem like a good plan either.

[EDIT] I want to add that I really would like to hear other suggestions for how to solve these kinds of issues. I could only imagine a truly no-consequences style working in a xerox parc-ish environment which is only possible when there's more than enough money (both on the corporate and the people side) so desperation isn't producing rash actions, and most people are being motivated by something other than the normal money/prestige.

I work at a company where On Call has become a monster. Week on, week off, no extra pay.

1) you get calls / emails from the clients. Anything from a P1 everything is on fire incident down to "we've seen some random SQL agent job has failed, drop what you are doing and give us an RCA now"

2) you get automated alerts via systems you dont own, like SQL Sentry, where someone somewhere years ago put in an alert that says "if XYZ batch job runs for 8 hours, alert" then has never touched the threshold since

3) you get automated alerts from systems you do own, which is a godsend because for once you can adjust down the noisy alerts

4) your manager or skip level will without warning create "dumb" (nuance free) Splunk alerts and expect you to see them, know when and how to respond to them minus any documentation to support the point of the alert or how to respond

5) your manager or skip level will accept any automated alert from any other dev or infra team and expect you to know when to respond, when to ignore, and don't you dare ask them to change the alerting thresholds to fix noise, that's not being a 'full stack on call'

6) you must respond to all of the business hours client email to the team distro, within 5 minutes of receipt. If someone puts SupportONcall@nolongerastartup.com on a thread, the subject instantly becomes your personal life mission until solved, dismissed by the email originator, or finally kills enough resources to annoy the manager to the point of (gasp) declaring the issue transient or not reproducible. Hope you like that your manager doubled the fields on JIRA tickets and marked them all as required.

7) everything in the company or business partners is in scope for your team until explicitly taken over by a dedicated team like DBAs

8) since we have one client with very strict SLAs, your manager has decided that now all of your alerts should be treated with equal urgency to those SLAs(response to an email within 30 minutes, 2 hr work around, 1 day fix)

In exchange for this, you get one work from home day per week, where you get to be online an extra hour on your designated day to be on call while the on call is in traffic home. That way, you are always responsive to email originators who cannot bear to wait until 6pm to get a response as to whether or not to worry about a missed backup or nolock-laden SQL select query that isn't working.

Somewhat exaggerated... But it's close enough that if you see this is deleted I probably am sitting in the discipline room or pink slipped.

At my last job I was offered a "promotion".

More responsibility, more accountability, overseeing junior staff AND on call. No roster, no clear definition of exactly what that would entail, but it was the kind of place that had thousands of system, and every day something was on fire.

All of that was offered for the glorious compensation rise of $0.

I happily turned down that "promotion" and it was clear the company hated me for it.

I like the two-tier approach although you need a large company to support it (yup, they do have upsides too).

While the product is still being developed/alpha/unstable, developers do the oncall. Benefit: they do have knowledge & having skin in the game works as motivation. This part is mentioned in the article, btw. But then, when the product matures, an SRE organization takes over.

Key point: they do so voluntarily and can request changes before taking over. This creates the good dynamic of separate people for separate roles, one can think of as 'judicial independence'. There's nothing like combining own skin in the game and the fact that you're pointing out deficiencies in somebody else's product (not yours) to get the extreme level of diligence typical for those reviews.

SRE review is a long process and generally assures the product adheres to a set of good practices surrounding monitoring, alerting, logging, playbooks, rollouts & canarying, emergency levers and whatnot.

I largely believe developers should be responsible for their work, which includes meeting the support requirements out of hours, and understanding the tooling and instrumentation required to keep their work healthy.

However, hiring people and not telling them this is part of the job is insanely unfair. People should get paid for being on-call. Further still, if something is going to wake me up during the night, I would expect to be able to work on those problems as a priority.

If I can't do that, then no dice, and thus this further explains why I don't really agree that Devs should be able to throw code over the wall at Ops, and ruin their lives whilst they sleep easy and priortise features over stability.