Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why you've never been in a plane crash

asteriskmag.com

Post reply on HN
Print view View original

71–80 of 310 posts

Re: Why you've never been in a plane crash

#71

Post by josephg »

EthicalSimilar wrote: post #52
izacus wrote: post #33

Earlier quoted context omitted.

TBH having software engineer output in general be liable to some minimum safety, correctness and quality standard would be a god send for the world. But of course the developers will revolt against that.

Would just cause more companies to outsource anyway.

It might do the opposite. Imagine if the company was held responsible for the quality of the software regardless of how it was made. I suspect it would be much easier to meet any quality standards if software was written in-house.

Re: Why you've never been in a plane crash

#72

Post by dustedcodes »

throwaway713 wrote: post #34

This concept of a blameless culture reminds me of one time when I was talking to a SWE at Facebook around 2010. I don’t know whether the story is actually true or just folklore, but apparently someone brought down the whole site on accident once, and it was pretty obvious who did it. Zuckerberg was in the office and walked up to the guy and said something along the lines of “Just so you are aware, it would probably t…

Whenever this happens to someone it's always a horrible feeling where one feels very guilty and ashamed no matter what people say to you and unfortunately mistakes like these are almost the bread and butter of any extremely experienced grey beard so it's kind of normal that something like this happens to someone at some point sooner or later. The only people who never make costly mistakes are those who were never trusted with responsibility in the first place.

So having said that I would like to emphasise that the cost which often gets quoted with those mistakes is not a real cost, it's an unrealised opportunity cost and sure it hurts, but you know what, the same company culture that allows such mistakes to happen and miss out on opportunity costs is the same culture which also allows engineers to quickly roll out important features and updates and therefore create more opportunity in the first place, and much faster as a whole, so in theory the cost doesn't come without the opportunity and it all evens itself out. Don't feel too bad about it.

Re: Why you've never been in a plane crash

#73

Post by latexr »

bheadmaster wrote: post #44
mulmen wrote: post #38

Earlier quoted context omitted.

Eh, that’s a really strange way to phrase it. Singling out the engineer isn’t blameless. Sure it’s a learning opportunity but it’s a learning opportunity for everyone involved. One person shouldn’t be able to take the site down. I have always thought of those situations as “failing together.”

> Singling out the engineer isn't blameless. Considering that everyone already knew who was responsible, I think saying "you won't be held accountable for this mistake" is the most blameless thing you can do. > Sure it's a learning opportunity but it's a learning opportunity for everyone involved. One person shouldn't be able to take the site down. The way I read the comment, it sounds to me exactly like what Zuckerb…

> Considering that everyone already knew who was responsible, I think saying "you won't be held accountable for this mistake" is the most blameless thing you can do.

What you’re describing isn’t blamlessness, it’s forgiveness. It’s still putting the blame on someone but not punishing them for it (except making them feel worse by pointing it out). Blamelessness would be not singling them out in any way, treating the event as if no one person had caused it.

> The way I read the comment, it sounds to me exactly like what Zuckerberg said.

Allegedly. Let’s also keep in mind we only have a rumour as the source of this story. It’s more likely that it never happened and this is a recounting of the Thomas Watson quote in other comments.

Re: Why you've never been in a plane crash

#75

Post by devjab »

throwaway713 wrote: post #34

This concept of a blameless culture reminds me of one time when I was talking to a SWE at Facebook around 2010. I don’t know whether the story is actually true or just folklore, but apparently someone brought down the whole site on accident once, and it was pretty obvious who did it. Zuckerberg was in the office and walked up to the guy and said something along the lines of “Just so you are aware, it would probably t…

When I was only a few years into my career I accidentally deleted all the Cisco phones in the municipality where I was a sowtware developer. I did it following the instructions of the IT operations guy in charge of them, but it was still my fault. My reaction was to go directly to the IT (who wasn’t my) boss and tell him about it.

He told me he wasn’t happy about the clean up they now needed to do, but that he was very happy about my way of handling the situation. He told me that everyone makes mistakes, but as long as you’re capable of owning them as quickly as possible, then you’re the best type of employee because then we can get to fix what is wrong fast, and nobody has to investigate. He also told me that he expected me to learn from it. Then he sent me on my way. A few hours later they had restored the most vital phone lines, but it took a week to get it all back up.

It was a good response, and it’s stuck with me since. It was also something I made sure to bring into my own management style for the period I was into that.

So I think it’s perfectly natural to react this way. It’s also why CEOs who fuck up have an easy time finding new jobs, despite a lot of people wondering why that is. It’s because mistakes are learning experiences.

Re: Why you've never been in a plane crash

#76

Post by latexr »

mbork_pl wrote: post #35
aramndrt wrote: post #9

Reminds me of the guy at GitLab, I believe it was, who accidentally and irreversibly deleted a bunch of repositories. GitLab had a fantastic approach to that whole ordeal and public embarrassment. They essentially blamed themselves for not having the proper contingencies in place instead of blaming the individual. It's worth reading up on.

Any source would be appreciated, I'd like to read more about it.

https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data...

From that link you should also be able to find conversations on Hacker News and the like. It was talked in a lot of places at the time.

If I’m remembering correctly (take this paragraph as imperfect memory), at the time a lot of people on the outside were looking to assign blame but the team tweeted something to the effect of “yes, we know who did it, and no, they won’t be fired” and didn’t even reveal who it was. Then they live-streamed the process of trying to recover as much as they could. They got a ton of community encouragement and it was widely viewed as the right way to handle things.

Re: Why you've never been in a plane crash

#77

Post by [deleted] »

bertil wrote: post #19
ssl232 wrote: post #12

One of the most important pieces of legislation in the UK that helped improve safety in workplaces was the Health and Safety at Work Act 1974, which placed a duty of care on organisations to look after the wellbeing of their staff. One of the tenets is to take near misses seriously. According to a health and safety engineer in a course I attended, near misses are like gifts from the heavens, as they show you exactly…

Very true. In general, I think that piece has a bit of a European blind side when starting by “The United States leads the world in airline safety.”

[deleted]

Re: Why you've never been in a plane crash

#78

Post by manojlds »

hkon wrote: post #62
manojlds wrote: post #54

Earlier quoted context omitted.

Strange way to describe the same situation and Zuck's thrust there in different words. Zuck is literally saying "failing together" and "learning together".

It's typically said to everyone at an all-hands meeting or similar, not to a single person. Delivering such a message personally to an individual is also sending another message.

It depends on the culture of the company.

Re: Why you've never been in a plane crash

#79

Post by 010101010101 »

siva7 wrote: post #63

>> It’s often much more productive to ask why than to ask who. In some industries, this is called a “blameless postmortem,” and in aviation, it’s a long-standing, internationally formalized tradition. If people die in an accident, blameless postmortem shouldn't be the answer but accountability. Otherwise events like the MCAS disaster would be just an "happy accident" with no one to blame.

I don’t think that’s what’s implied here at all - the focus on blamelessness is to get to the root cause of failure and prevent future occurrences rather than to punish existing failures without affecting any actual change.

Re: Why you've never been in a plane crash

#80

Post by manojlds »

mulmen wrote: post #50
bheadmaster wrote: post #44

Earlier quoted context omitted.

> Singling out the engineer isn't blameless. Considering that everyone already knew who was responsible, I think saying "you won't be held accountable for this mistake" is the most blameless thing you can do. > Sure it's a learning opportunity but it's a learning opportunity for everyone involved. One person shouldn't be able to take the site down. The way I read the comment, it sounds to me exactly like what Zuckerb…

In a blameless culture there is no one person responsible. It should not have been possible for one person to take the site down. There should have been controls in place to prevent it. There shouldn’t even be someone that “everyone knows” is responsible, because by definition that’s impossible.

What part of "so let’s just consider it an expensive learning opportunity to redesign the system so it can’t happen again" doesn't mean that it's happened, but let's see how we get there.

"It should not have been possible for one person to take the site down" - yes, and that's exactly what Zuck is addressing here? May be such controls are there across the development teams and some SRE did something to bring it down and now there needs to be even better controls in that department as well?

Post reply on HN