Why Heroism Is Bad and What We Can Do to Stop It • Hacker News Board

Re: Why Heroism Is Bad and What We Can Do to Stop It

#41

Post by tithe » Tue, Aug 06, 2024, 6:53 PM UTC

andrewla wrote: post #40

tithe wrote: post #39

Earlier quoted context omitted.

I do agree. If persistent heroic acts are a requirement to keep any system running, then let's allocate resources to determine why that's the case and how the system can be changed into something more resilient.

The problem is that it sounds good to talk about "allocating resources" and "communication" and making the system more resiliant. But even identifying places where heroism is being applied is extremely difficult except for the hero. Most importantly, the people who need to know that herois is keeping things alive will often be completely unaware that there is even a problem. In the end the only way to do this is to a…

> In the end the only way to do this is to allow the system to fail.

This may be the efficient way for systems under test, but for a live, production system there must be higher bar of performance than "let it fail". I agree with several of the points Malmberg makes (which my original sarcastic comment probably doesn't suggest), but his final conclusion of "let the system break" is alarming and dangerous.

> If you find yourself in a position of being a hero, you have to notice it and do something about it.

If I found myself being the hero, I would absolutely push this forward and do something about it. It's also a tragedy that this may actually result in the opposite outcome that you want (like being fired for "not being a team player"). At the end of the day, it's still human beings in charge of these sytems, which means handling our communications with grace and tact.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#42

Post by saddist0 » Tue, Aug 06, 2024, 7:50 PM UTC

Swizec wrote: post #3

In my experience being the hero is fantastic … until you want to go on vacation, have a sick day, change teams, or get promoted. Hard to promote someone irreplaceable. Always code (or mentor) yourself out of the job and let others play with your legos. Even if they do it wrong.

Watch the world burn with vodka in hand. It's much better than doing nothing while you can.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#43

Post by saddist0 » Tue, Aug 06, 2024, 7:57 PM UTC

This article highlights many pitfalls but fails to explain "how to practice heroism effectively".

For instance, a team member might notice a recurring pattern and repeatedly save the SLA by addressing it immediately. While this quick fix is heroic, it should also be escalated for a long-term solution. This way, the hero tackles the immediate issue, and the team ensures that such heroism isn't needed in the future, and so on.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#44

Post by ASalazarMX » Tue, Aug 06, 2024, 8:57 PM UTC

tithe wrote: post #39

andrewla wrote: post #35

Earlier quoted context omitted.

Being the hero once is fine -- but if the only reason we don't have outages on every update is because of an SRE who takes it upon themselves to babysit the update, then the system is super broken and there will come a time when that babysitting doesn't work and the lack of awareness of the problem will cause the cascade to be that much worse.

I do agree. If persistent heroic acts are a requirement to keep any system running, then let's allocate resources to determine why that's the case and how the system can be changed into something more resilient.

I also concur. A firefighter putting out a dangerous kitchen fire is heroic, but putting out the same fire several times without finding out the cause is negligent.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#45

Post by andrewla » Tue, Aug 06, 2024, 9:23 PM UTC

tithe wrote: post #41

andrewla wrote: post #40

Earlier quoted context omitted.

The problem is that it sounds good to talk about "allocating resources" and "communication" and making the system more resiliant. But even identifying places where heroism is being applied is extremely difficult except for the hero. Most importantly, the people who need to know that herois is keeping things alive will often be completely unaware that there is even a problem. In the end the only way to do this is to a…

> In the end the only way to do this is to allow the system to fail. This may be the efficient way for systems under test, but for a live, production system there must be higher bar of performance than "let it fail". I agree with several of the points Malmberg makes (which my original sarcastic comment probably doesn't suggest), but his final conclusion of "let the system break" is alarming and dangerous. > If you fi…

Just from experience, the system failures tend not to be of this mode. If heroism is routinely deployed, then yes, failures can be huge and even more heroism then needs to be applied.

But normally, what failures look like is a degredation in responsiveness or a failure to scale up quickly enough for surging demand or faster turnaround on canary failures or caches that need to be purged after batch jobs, etc.

Much more "degredation below SLA" rather than "every windows machine in the world blue screens". Heroism for disasters like that, sure, but that's going to be a post-mortem and a big deal. Most of the time failures are small, and letting them fail means that generally there is more awareness of the problem -- clearing caches or restarting the instances because they get slow conceals the problem and become part of the background routine.

The note on heroism is not a note for managers -- it's a note for SREs to actively notice when they are engaging in heroism and to stop doing that. Letting the cache get overloaded so that an automated system can do the purges because the development team is now aware of the issue is far preferable. And sometimes these routine acts of heroism become routine process/superstition to fix problems that no longer exist, or that are minor and not worth the time spent.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#46

Post by lutarezj » Wed, Aug 07, 2024, 5:48 AM UTC

RHSeeger wrote: post #19

lutarezj wrote: post #10

Not sure if heroism is bad. [1] All teams should have a Jordan, a kobe, a shaquille or a combi. One needs A players and supporting cast. It is not the culture or the org who decides upon the evolution of the heroism. It is the hero who builds a team around him/her. [2] the scrum or agile saga that promotes that all team members should be able to do what all team members do is just excel-minded-nonesense. Cant win cha…

What you're describing isn't at all what the linked page is discussing.

Agreed. The other plausible, realistic and decent option is that the author of the reaction to my comment does not understand the article at all and does not understand much about who has influence in a company and who’s incentivized to hide, change or create problems. Good luck!

Re: Why Heroism Is Bad and What We Can Do to Stop It

#47

Post by xtiansimon » Wed, Aug 07, 2024, 12:27 PM UTC

What a bombastic title—heroism is bad.

I’m thinking this whole piece is slanted to correct some other toxic or difficult to manage culture issue.

Getting to examples quickly saves the piece. Sounds like there are some gung-ho youths happy to be working at Google and they need some mentoring.

Re: Why Heroism Is Bad and What We Can Do to Stop It

#48

Post by arccy » Sat, Aug 10, 2024, 9:25 PM UTC

chrchr wrote: post #37

The last slide says "let the system break". I strongly suggest that after that slide, there needs to be a whole series of slides about how to make it so that it's ok to let the system break. If you haven't already done the hard work to make your stuff resilient, "let the system break" is a recipe for blowing up customers, damaging reputations, and hurting people.

sometimes that's the only way to get the message across to leaders

Re: Why Heroism Is Bad and What We Can Do to Stop It

#49

Post by cafard » Sun, Aug 11, 2024, 1:16 AM UTC

automatoney wrote: post #11

JohnMakin wrote: post #6

There are people so addicted to this that they will literally create problems out of nowhere so they can pull some heroics and save the day, always with high visibility from management. Seen a person advance pretty far in their career this way. Until management stops incentivizing this behavior, it won't stop. This is a management issue - which seems weird because this writing seems targeted towards IC's.

I haven't run into this person but I'd love to hear about problems you've seen people create - I can't even think of a problem I could create and fix.

Have you never heard of a volunteer firefighter starting fires? It doesn't happen often, but it does happen. (And maybe with professional firefighters, I don't know.)

Re: Why Heroism Is Bad and What We Can Do to Stop It

#50

Post by hnthrowaway0315 » Sun, Aug 11, 2024, 3:07 AM UTC

Swizec wrote: post #3

In my experience being the hero is fantastic … until you want to go on vacation, have a sick day, change teams, or get promoted. Hard to promote someone irreplaceable. Always code (or mentor) yourself out of the job and let others play with your legos. Even if they do it wrong.

That's what I precisely learned from things such as Bank Python:

1. Create legos

2. Jump to another similar place to create the same legos