Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A few words about Blameless culture

gybe.ca

Post reply on HN
Print view View original

41–50 of 52 posts

Re: A few words about Blameless culture

#41

Post by pclmulqdq »

Grambles wrote: post #40
pclmulqdq wrote: post #38

Earlier quoted context omitted.

I understand the point of blameless postmortems, thank you. I am suggesting that Americans in particular take it too far, to the point of excusing negligent behavior (which you should be trying to prevent) rather than just avoiding punishing people for mistakes. At the same time, we have made the words "you screwed up" almost a swear in the workplace. If you can normalize "you screwed up" so that everyone hears it (a…

Using the literal words: "You screwed up." -- can you make an example of a way that would be helpful in an incident, either during the process, or after? I can't. There's no value in it. What did we, as a team, do that allowed the incident to happen? Yes, John Smith shouldn't have dropped the tables on production, obviously, but does he really not know that as part of the incident response that he's (presumably) also…

I can, in that given situation. Your assumption is that John Smith should be told "you screwed up" for dropping the prod tables. He shouldn't - he made a normal mistake. The negligent party here, who should hear the words "you screwed up," is Johnny's boss or tech lead, who decided that those tables should be droppable in the first place, despite the obvious risk.

"Negligent" doesn't just mean "made a mistake." It means something more like "their carelessness led to a mistake."

That person hearing "you screwed up" will cause significant behavior change. I daresay it will encourage them to make the prod tables very hard to drop, and since they are presumably a smart person, when combined with the postmortem of the incident, it will encourage them to look for and proactively fix similar problems, and generally align the team with good DevOps practices.

It is important in all of this that the right person gets the message. I assume you expect that to not happen, since that is one of the theses of "blameless postmortems."

Re: A few words about Blameless culture

#42

Post by Grambles »

pclmulqdq wrote: post #41
Grambles wrote: post #40

Earlier quoted context omitted.

Using the literal words: "You screwed up." -- can you make an example of a way that would be helpful in an incident, either during the process, or after? I can't. There's no value in it. What did we, as a team, do that allowed the incident to happen? Yes, John Smith shouldn't have dropped the tables on production, obviously, but does he really not know that as part of the incident response that he's (presumably) also…

I can, in that given situation. Your assumption is that John Smith should be told "you screwed up" for dropping the prod tables. He shouldn't - he made a normal mistake. The negligent party here, who should hear the words "you screwed up," is Johnny's boss or tech lead, who decided that those tables should be droppable in the first place, despite the obvious risk. "Negligent" doesn't just mean "made a mistake." It me…

I suppose I agree more with you than before, but I still think that aside from the fact that a manager or tech lead is ostensibly used to hearing people be angry (or frustrated, or whatever) -- why is John's boss more deserving of "You screwed up." than the person who dropped them? Yeah, obviously, the prod tables shouldn't have been droppable. John still shouldn't have dropped them. In fact, John, the original developer who implemented them, anyone who altered them since, the tech lead, manager, and frankly anyone who knew this data was critical could have all raised the alarm.

Ideally a blameless post mortem allows the freedom to identify any of the potential fixes that could've stopped this, and empowers anyone who could've dealt with it to deal with future issues. If you blame the manager then that can implicitly absolve everyone else in the chain.

With that said, I would agree that having a primary owner of things does matter. For that reason, sure, making the manager more aware might help in future. I still think it's a bad idea for org culture though because many managers will respond to "You screwed up." with trying to ensure future blameables find their way to another target. Instead, I'd prefer approaching the manager with "We could've caught this in [any of the ways we could've caught it].", and if the manager doesn't care at that point they're just fully incompetent.

Re: A few words about Blameless culture

#43

Post by resonious »

lazyasciiart wrote: post #36
resonious wrote: post #23

This "blameful, not blameless" wording is a bit confusing to me. Are we blaming or not? "Blameful" isn't even a word, according to my web browser. The browser is showing red squiggles and so is my brain. I think it'd be a little more potent to just say hey, there's a difference between saying someone is partially responsible for an issue, and pestering them about it (e.g. telling them they did a bad thing when they l…

pestering them? What people are actually worried about is firing them, which still happens even at companies that pretend they know better. I heard from friends at Salesforce that this guy was fired, for instance. https://www.theregister.com/2021/05/19/salesforce_root_cause...

I'll agree that's bad, but I'd also say that blaming != firing.

Re: A few words about Blameless culture

#44

Post by seadan83 »

resonious wrote: post #43
lazyasciiart wrote: post #36

Earlier quoted context omitted.

pestering them? What people are actually worried about is firing them, which still happens even at companies that pretend they know better. I heard from friends at Salesforce that this guy was fired, for instance. https://www.theregister.com/2021/05/19/salesforce_root_cause...

I'll agree that's bad, but I'd also say that blaming != firing.

Though, blame might be quite career limiting

Re: A few words about Blameless culture

#45

Post by seadan83 »

pclmulqdq wrote: post #16
ChrisMarshallNY wrote: post #15

> He very likely has no idea how much impact he had on my career, but I’m very thankful for his patience with me that day. There have been many folks like this, in my career. I am humbled and grateful. I worked for a Japanese corporation, where they didn't "blame," per se , but did figure out if someone was Responsible. If so, that someone was expected to cop to it, and the team would then close ranks, and fix the is…

I do sometimes wonder if the (American) culture of blamelessness has led to a general lack of a feeling of responsibility among software engineers, leading to general carelessness. The idea of responsibility that becomes opaque as you move up the hierarchy sounds good, but I also expect that also has a lot to do with Japanese culture.

I don't know if there is an American culture of blamelessnes. Lots of variation, such a blanket statement needs more than confirmation bias.

With that said, I think your comment describes kiss-up and corporate fiefdom culture. There are plenty of examples of that. I wonder if non-American cultures are different in that respect.

One very key distinction is freedom to admit error vs "the baby is not ugly." I don't think blameless cultures are the norm, where people volunteer to write post portems and voluntarily submit after action plans..

Re: A few words about Blameless culture

#46

Post by pclmulqdq »

Grambles wrote: post #42
pclmulqdq wrote: post #41

Earlier quoted context omitted.

I can, in that given situation. Your assumption is that John Smith should be told "you screwed up" for dropping the prod tables. He shouldn't - he made a normal mistake. The negligent party here, who should hear the words "you screwed up," is Johnny's boss or tech lead, who decided that those tables should be droppable in the first place, despite the obvious risk. "Negligent" doesn't just mean "made a mistake." It me…

I suppose I agree more with you than before, but I still think that aside from the fact that a manager or tech lead is ostensibly used to hearing people be angry (or frustrated, or whatever) -- why is John's boss more deserving of "You screwed up." than the person who dropped them? Yeah, obviously, the prod tables shouldn't have been droppable. John still shouldn't have dropped them. In fact, John, the original devel…

I will point out that in the ideal scenario you're talking about, there is probably a next sentence after "at that point, they're just fully incompetent." That next sentence involves some real-life consequences of that incompetence. At that point, the postmortem is not really blameless, the blame is just one step removed.

I agree that what you are describing is the ideal, but I don't think it happens often. What I have seen more is that the postmortem turns into 10 different open bugs, 5 of which (usually bandaids that stop that particular failure mode) get closed quickly and the rest of which (often including the real solution) get put off until the next crisis.

Ownership is really important, but hand in hand with ownership is responsibility. That step is missing in many orgs today.

Re: A few words about Blameless culture

#47

Post by PH95VuimJjqBqy »

meowfly wrote: post #34
PH95VuimJjqBqy wrote: post #29

Earlier quoted context omitted.

but what does 'done' mean here? quality is a thing and forward movement with quality is possible.

Quality is obviously a thing your team should care about, but quality is also a process problem. If an individual continually produces poor quality than the "process" should fix it via not merging their code or pip or something else. What we are avoiding here is a CEO asking, "What caused the outage" and your manager responds "Bill wrote some code that caused a deadlock". It's possible Bill is working on fragile part…

quality is not a process problem, process is generally put into place to protect against bad decisions which often also constrains good decisions (because it typically does so by constraining decisions).

avoidance of negative quality is not quality.

Re: A few words about Blameless culture

#48

Post by grvdrm »

nonrandomstring wrote: post #5

Really enjoyed this too. Most of all the author's gratitude toward the patient fire chief whose leadership example he praises. We could do with more of this in cybersecurity. We're all surrounded by insecure software and defective devices churned out by companies chasing a quick profit. We're mired in complexity, non-determinism and broken systems. The experts who designed them throw up their hands in confusion. And…

I'd love to chat w/you about this re: cyber. I am at a software vendor in the incident response space. It's always good to know other folks in the industry and collaborate on ways to help improve what I'll rephrase as "the human element" in the broader cyber space.

Should I DM you from info on your site?

PS: your website is so great. Love it.

Re: A few words about Blameless culture

#49

Post by chfalck »

galangalalgol wrote: post #20
SilkRoadie wrote: post #18

I like the article and I have been promoting a blameless culture for most of my career. To respond rapidly to failings, we need transparency about what has happened. The same is required to ensure it doesn't happen again. I have seen a lot of incidents and I cannot think of one where a single person was to blame. Sure, one person ran a command or made a bad commit. However, someone else granted them access, someone t…

I have worked years in a culture that attributes all mistakes to defects in the process. And the only possible individual mistake is not to follow the process. Process culture is just as bad as blame culture in some ways. It leads to ever more complex processes. The teams I've seen do best, the manager of the project will allow process to be informal and treated as best practices only. Then people are held accountabl…

Process culture can be nice in the ideal scenario where the processes can have huge parts automated or programmatically guard railed. However, when that isn’t the case, I agree the paperwork barriers can get dreadfully tedious and introduce their own opportunities for mistakes

Re: A few words about Blameless culture

#50

Post by nonrandomstring »

grvdrm wrote: post #48
nonrandomstring wrote: post #5

Really enjoyed this too. Most of all the author's gratitude toward the patient fire chief whose leadership example he praises. We could do with more of this in cybersecurity. We're all surrounded by insecure software and defective devices churned out by companies chasing a quick profit. We're mired in complexity, non-determinism and broken systems. The experts who designed them throw up their hands in confusion. And…

I'd love to chat w/you about this re: cyber. I am at a software vendor in the incident response space. It's always good to know other folks in the industry and collaborate on ways to help improve what I'll rephrase as "the human element" in the broader cyber space. Should I DM you from info on your site? PS: your website is so great. Love it.

I'd love to chat about that. Organisational psychology is fascinating, but I'm no expert, just " a reader" as Bill Hicks would say.
Post reply on HN