I like the expression "Bad News Fast": it gets the message across very efficiently, and if it's correctly incentivized (even just verbal praise) it can be a powerful tool to have an accurate view of the situation.
A few words about Blameless culture
11–20 of 52 posts
Re: A few words about Blameless culture
#12Really enjoyed this too. Most of all the author's gratitude toward the patient fire chief whose leadership example he praises. We could do with more of this in cybersecurity. We're all surrounded by insecure software and defective devices churned out by companies chasing a quick profit. We're mired in complexity, non-determinism and broken systems. The experts who designed them throw up their hands in confusion. And…
I was hoping to read he'd reached out to the fire chief though. Praise is never given enough.
Re: A few words about Blameless culture
#13Practising the workflow itself can be helpful too.
Some years ago, we had an "incident" where one of our developers discovered he had admin access even though he was logged into a different account of his. It was a multi-tenant SaaS. Since he was supposed to be admin anyway, I wondered where the harm was.
Now, though, I think he was just practising the blameless incident response in case something serious did eventually come up. Once that happens, you don't have time to practise in peace anymore.
Re: A few words about Blameless culture
#14"The org’s leadership was clear that the individual was not fired for making the mistake, but for the attempted coverup." Somehow I have a hard time believing that. The author using "individual" in prose is a give away for militaristic leadership style, where you quickly learn to cover up any mess up so the direct superiors don't lose face. Blameless ...
True, it still fails the game theory test. You should still assume the need to cover up, because it might work, and in a (common) poor intitutional culture, save you from reprimand/job loss. Worst case, you are found out, and fired, but you had the chance to hide or obfuscate the fault. In a common blameful culture, you may have been blamed simply for (being seen as) the one causing the fault, whether volunteered or hidden.
If you cover up in a genuinely blameless root cause analysis but are found out, the blameless culture would see that as a systemic fault (them not communicating the importance of blameless root cause, like the fire chief of the example), and if not found out, it is merely a less effective analysis. It is an unfortunate upfront cost imposed by the proponderance of BF cultures on BL ones.
Like a market for lemons, it means you should assume CYA until a blameless culture is clearly and overtly demonstrated, not merely stated.
Re: A few words about Blameless culture
#15> He very likely has no idea how much impact he had on my career, but I’m very thankful for his patience with me that day.
There have been many folks like this, in my career. I am humbled and grateful.
I worked for a Japanese corporation, where they didn't "blame," per se, but did figure out if someone was Responsible. If so, that someone was expected to cop to it, and the team would then close ranks, and fix the issue. Lesson learned.
Re: A few words about Blameless culture
#16> He very likely has no idea how much impact he had on my career, but I’m very thankful for his patience with me that day. There have been many folks like this, in my career. I am humbled and grateful. I worked for a Japanese corporation, where they didn't "blame," per se , but did figure out if someone was Responsible. If so, that someone was expected to cop to it, and the team would then close ranks, and fix the is…
I do sometimes wonder if the (American) culture of blamelessness has led to a general lack of a feeling of responsibility among software engineers, leading to general carelessness. The idea of responsibility that becomes opaque as you move up the hierarchy sounds good, but I also expect that also has a lot to do with Japanese culture.
Re: A few words about Blameless culture
#17> He very likely has no idea how much impact he had on my career, but I’m very thankful for his patience with me that day. There have been many folks like this, in my career. I am humbled and grateful. I worked for a Japanese corporation, where they didn't "blame," per se , but did figure out if someone was Responsible. If so, that someone was expected to cop to it, and the team would then close ranks, and fix the is…
I do sometimes wonder if the (American) culture of blamelessness has led to a general lack of a feeling of responsibility among software engineers, leading to general carelessness. The idea of responsibility that becomes opaque as you move up the hierarchy sounds good, but I also expect that also has a lot to do with Japanese culture.
The US has a very strange cultural approach to apologizing and accepting Responsibility.
I know that part of it is lawyers and police. If you say "I'm sorry," in many contexts, that puts you on the hook, legally. Most lawyers will advise you to keep your mouth shut, and never, ever admit a thing.
I was taught to promptly admit when I'm wrong; regardless of the consequences. Also, to not cop to stuff that I'm not responsible for.
I have often had arguments with (American) people, where they are 100%, without-a-shred-of-doubt, wrong. Often, embarrassingly so, yet, they Will. Not. Admit. It.
I could bring the spirits of Carl Sagan and Stephen Hawking in, to show them they are wrong, and they will still never admit it.
It's actually kind of sad.
My ability to apologize worked extremely well, in the Japanese corporation. They respected it, and I was given a great deal of deference.
On the other hand, the Americans tended to be pretty much dicks about it. If I apologized, they usually tried to use it to gain some leverage, and bully me.
Re: A few words about Blameless culture
#18I like the article and I have been promoting a blameless culture for most of my career. To respond rapidly to failings, we need transparency about what has happened. The same is required to ensure it doesn't happen again.
I have seen a lot of incidents and I cannot think of one where a single person was to blame. Sure, one person ran a command or made a bad commit. However, someone else granted them access, someone trained them. A manager either reviewed the process performed or never considered a process was required. A lot of company cultures do not promote proper risk management in technical processes.
I get a lot of satisfaction from technical post-mortems. There is always something that could have been done, a process that could have been in place or a software/infrastructure change to mitigate the problem.
A couple of companies have worked for have had individuals that did not subscribe to this culture. They would want a name. I never knew why exactly. Maybe it was to block a pay bump or defer a promotion. As the tech lead or dev manager any team failings are my responsibility and in companies where bullets are fired I take them. I find this protects the team and helps a blameless culture thrive amongst engineers.
Generally, I find this culture leads to fewer incidents and problems as the openness when things are going wrong allows for faster response times and software/process changes in review.
Re: A few words about Blameless culture
#19Earlier quoted context omitted.
I do sometimes wonder if the (American) culture of blamelessness has led to a general lack of a feeling of responsibility among software engineers, leading to general carelessness. The idea of responsibility that becomes opaque as you move up the hierarchy sounds good, but I also expect that also has a lot to do with Japanese culture.
The US has a very strange cultural approach to apologizing and accepting Responsibility. I know that part of it is lawyers and police. If you say "I'm sorry," in many contexts, that puts you on the hook, legally. Most lawyers will advise you to keep your mouth shut, and never, ever admit a thing. I was taught to promptly admit when I'm wrong; regardless of the consequences. Also, to not cop to stuff that I'm not resp…
Inability to admit wrongdoing is just one of many diagnostic criteria for antisocial personality (sociopaththy). In my experience its only mid level managers and up that are explicitly trained not to admit wrongdoing in the US. I've always felt free to do so and all the good engineers I've worked with have done the same.
Re: A few words about Blameless culture
#20I like the article and I have been promoting a blameless culture for most of my career. To respond rapidly to failings, we need transparency about what has happened. The same is required to ensure it doesn't happen again. I have seen a lot of incidents and I cannot think of one where a single person was to blame. Sure, one person ran a command or made a bad commit. However, someone else granted them access, someone t…
I have worked years in a culture that attributes all mistakes to defects in the process. And the only possible individual mistake is not to follow the process. Process culture is just as bad as blame culture in some ways. It leads to ever more complex processes. The teams I've seen do best, the manager of the project will allow process to be informal and treated as best practices only. Then people are held accountable for bad results by putting them on tasks that can cause less damage. Good results mean more critical tasks. No single mistake kills a performance review or promotion, or makes one. We all make mistakes and get lucky. But over time reapeated mistakes or successes do have an impact.