Live data from Hacker News

Why Heroism Is Bad and What We Can Do to Stop It

sre.google

31–40 of 50 posts

Re: Why Heroism Is Bad and What We Can Do to Stop It

#32

You don't want heroes at large companies with top down product management. You need heroes at small innovative startups. This write up is more of a documentation on the stagnant culture inside google

Yeah, the whole thing reads kind of like a research paper "Amazing Procedure Dramatically Improves Group Cohesion - - in mice."

Analogous to "Employees Having Outstanding Problem-Solving Initiative; Shown Detrimental to Company Objectives - - at Google."

Re: Why Heroism Is Bad and What We Can Do to Stop It

#33
post #4
post #2

Heroism is what you do until you manage to secure the headcount and hire the team that lets you run things smoothly. In the real world, getting approval for headcount can take 6 months, hiring 3, training another 3. So you need to sustain heroism for a year without burning out.

Heroism as described in this slideshow prevents anyone from realizing/accepting that they need to hire the appropriate headcount.

While not doing heroism prevents your product/service from being successful or reach the market in a timely fashion.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#34
Did any of the commenters read the slideshow? Heroism is bad when it covers systemic problems.

Heroes are great -- SREs who rise to the occasion to prevent horrors are appropriately rewarded and congratulated for their work.

But when a product relies upon heroes to continue operating, you are in a dangerous situation. That's how major outages occur; the hero goes on vacation or decides to let it break this time and the cascade of failures causes huge amounts of damage, where letting the system break much earlier would have made it clear to the development team that there is a major gap in the intrinsic reliability of the system.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#35
post #21

I imagine a conversation with this individual as team lead would go something like this: "So, you worked overtime to save systems across the planet from crashing due to a botched update?" "Yes, sir. We're 'Site Reliability Engineering', after all." "And people in airports don't have to sleep on the floors because airlines can actually schedule flights?" "Yes, sir. Site Reliability Engineering, at its finest, sir!" "N…

Being the hero once is fine -- but if the only reason we don't have outages on every update is because of an SRE who takes it upon themselves to babysit the update, then the system is super broken and there will come a time when that babysitting doesn't work and the lack of awareness of the problem will cause the cascade to be that much worse.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#36

You don't want heroes at large companies with top down product management. You need heroes at small innovative startups. This write up is more of a documentation on the stagnant culture inside google

You DO need heroes at large companies with top down product managers.

But even small companies cannot afford to have heroism be the only reason that their systems work.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#37
The last slide says "let the system break".

I strongly suggest that after that slide, there needs to be a whole series of slides about how to make it so that it's ok to let the system break. If you haven't already done the hard work to make your stuff resilient, "let the system break" is a recipe for blowing up customers, damaging reputations, and hurting people.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#38
post #8

"No matter how many hours they need to work." "No matter that they need to work evenings and weekends." I don't call these people heroes, I call them idiots. Also, not being able to copy/paste text from text slides is a pretty terrible design choice, but we shouldn't be surprised knowing what the source is.

"I'm going to be a Happy Idiot

and struggle for the Legal Tender

where the Ads Take Aim

and Lay Their Claim

to the Heart and the Soul of the Spender

And believe in Whatever May lie

in those things that Money Can Buy

though True Love could have been a Contender

Are you there?

Say a Prayer

for The Pretender

who started out so Young and Strong

only to Surrender"

- - J. Browne, 1976

Re: Why Heroism Is Bad and What We Can Do to Stop It

#39
post #21

I imagine a conversation with this individual as team lead would go something like this: "So, you worked overtime to save systems across the planet from crashing due to a botched update?" "Yes, sir. We're 'Site Reliability Engineering', after all." "And people in airports don't have to sleep on the floors because airlines can actually schedule flights?" "Yes, sir. Site Reliability Engineering, at its finest, sir!" "N…

Being the hero once is fine -- but if the only reason we don't have outages on every update is because of an SRE who takes it upon themselves to babysit the update, then the system is super broken and there will come a time when that babysitting doesn't work and the lack of awareness of the problem will cause the cascade to be that much worse.

I do agree. If persistent heroic acts are a requirement to keep any system running, then let's allocate resources to determine why that's the case and how the system can be changed into something more resilient.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#40
post #39

Earlier quoted context omitted.

Being the hero once is fine -- but if the only reason we don't have outages on every update is because of an SRE who takes it upon themselves to babysit the update, then the system is super broken and there will come a time when that babysitting doesn't work and the lack of awareness of the problem will cause the cascade to be that much worse.

I do agree. If persistent heroic acts are a requirement to keep any system running, then let's allocate resources to determine why that's the case and how the system can be changed into something more resilient.

The problem is that it sounds good to talk about "allocating resources" and "communication" and making the system more resiliant. But even identifying places where heroism is being applied is extremely difficult except for the hero. Most importantly, the people who need to know that herois is keeping things alive will often be completely unaware that there is even a problem.

In the end the only way to do this is to allow the system to fail. If you find yourself in a position of being a hero, you have to notice it and do something about it. You could do a big writeup of how to fix the system to remove the requirement for heroism, etc., but as an SRE you don't always have insight into how important the particular issue is, so you could be wasting a bunch of time on something completely unnecessary (or that is not worth your time or the time to fix it).

Post reply on HN