Live data from Hacker News

It's about what broke, not who broke it

rachelbythebay.com

81–90 of 105 posts

Re: It's about what broke, not who broke it

#81
post #53

I work in the nuclear industry, where most places are pretty good about maintaining a "blame-free" culture. You focus on what processes and procedures failed, what controls were missing, etc., that allowed somebody to make a mistake. As this attitude was adopted, things shifted too far (at least in the opinion of industry groups, and my observation) to the point where people underperforming to the point of negligence…

> where people underperforming to the point of negligence

Well (to play devil's advocate just a bit) - isn't the ultimate end-goal of an end-state robust process one in which people can not just underperform, but be replaced completely?

Re: It's about what broke, not who broke it

#82

Earlier quoted context omitted.

Demoralizing - why? That seems like an attitude problem on the part of “Charlie”, not “Bob”. If “Charlie” is going to slink off with his tail between his legs every time he makes a mistake, he’ll have a tough time of it - it’s not like everyone can’t SEE that he broke it through version control anyway! I just don’t really get it. Even when I was a junior, if I overheard “this thing is broken,” I was the first to pop…

I’m with you 100% except you’re not taking into account what I said about this being in public and Alice not being a part of the project. Internally assigning blame isn’t the issue, it’s about the “team” facade being shattered when dealing with the outside. If you’ve accepted Charlie into the organization then from without it isn’t about Charlie or Bob, the answer should be “yes, we’re aware; a recent commit broke th…

I often compare open source projects based on what's visible on the github page.

Drama around a volunteer team in the open is a bad smell.

Edit: I'd like to explain why.

* Open source projects with lots of drama often don't attract new talented developers, and if talent happens to depend on that codebase they are more likely to fork and start a new community, or fork and not submit pull requests.

* If I need to interact with the team for pull requests or bug/support tickets I'd like to feel assured we can do so respectfully and professionally.

* If a community has drama in it I am less likely to recommend the software to a friend or blog about the software because I won't want to be associated with it. I'm more likely to stop using it and switch to a different solution.

Re: It's about what broke, not who broke it

#83

Earlier quoted context omitted.

Demoralizing - why? That seems like an attitude problem on the part of “Charlie”, not “Bob”. If “Charlie” is going to slink off with his tail between his legs every time he makes a mistake, he’ll have a tough time of it - it’s not like everyone can’t SEE that he broke it through version control anyway! I just don’t really get it. Even when I was a junior, if I overheard “this thing is broken,” I was the first to pop…

Two things: First, it's generally best to praise publicly, and criticize in private. Second, saying "@Users's commit screwed-the-pooch" blames but, frankly, may not be the whole picture. It's entirely possible that the commit caused the issue, but everything was done by the book in which case it's really an organizational failure. Personally, I sympathize with your argument. I have no personal problem with Torvald-st…

Right. Is the codebase spaghetti? Did anybody take the time to help Charlie understand the system? Were there unit tests that should've been broken by Charlie's commit? Were there unit tests but no continuous integration, so he didn't know to run those particular tests? Was there no code review where somebody could've caught the bug? Was there a code review but nobody else caught the bug? Was there QA testing performed where the bug could've been caught?

Etc. etc.

Re: It's about what broke, not who broke it

#84
post #29

When someone makes a mistake, that's an incredible investment in them. I'm always surprised* when people try to throw it away by firing them or making them want to quit. Help them learn from it and apply that knowledge moving forward. Otherwise they're just taking that knowledge and using it to help another company. *Obviously with the caveat that some people are repeat offenders who are careless or just not good emp…

In other professions some mistakes cost the professional real money (doctor malpractice) or cause them to lose their license (drinking and driving with a commercial vehicle license).

As an industry we don't have a response to a truly neglectful mistake yet.

Re: It's about what broke, not who broke it

#85
post #67

Reminds me of when someone ran "rm -rf /" at Pixar and deleted all of Toy Story 2. The backups were crap and the only reason it survived was because someone took a server to work from home. When all was said and done, they never really found who did it, they just made organisational changes to ensure it didn't happen again. No blame game.

When I worked with my first non-remote team in Phoenix, I basically did this to our mobile app codebase with an in-house git repository due to some faulty rsync changes to a grunt task.

To the old NPL team, sorry about that. Culture is important.

Re: It's about what broke, not who broke it

#86

I do a lot of open source work and unfortunately a very common posion is focusing on “who broke it,” which is especially disparaging when done in public. A particularly nasty habit is when outsider Alice opens an GitHub issue saying “xxxx is broken” and developer Bob replies with “yup, @Charlie’s commit fubar’d everything.” Unfortunately both very demoralizing and very common.

I wonder, though, how much the culture of "talent" and "rockstar developers" contributes to this. We programmer types often perpetuate this narrative that programming ability is something that you're "born with" and you have it or you don't - unlearnable, unteachable and ephemeral is the mystique of the programmer. So, how do you figure out who the capable ones are and who the incapable ones are? Well, of course - the ones who f'ed something up are the incapable ones, who just didn't "have it" after all.

Re: It's about what broke, not who broke it

#87
post #78
post #57

Earlier quoted context omitted.

> [...] not all settings were applied until you opened the UI provided by the vendor. [...] the NICs would reboot, just long enough to kill TCP connections. The UI part suggests that it was Windows, and if it was, it's not quite the case that "just long enough" to kill TCP connections, as you need quite a lot of downtime to terminate a typical TCP session. In Windows, if a NIC goes down, all the TCP connections that…

Windows is often the only vendor-supported choice for interfacing your computer applications to PLCs and such things. Also most of the proprietary protocols run over industrial ethernet are some kind of legacy serial (232, 485..) bytestream format wrapped in TCP and the software usually does not handle loss of the TCP connection particularly gracefully. (on multiple occasions I've seen rules like "reboot the whole in…

> Windows is often the only vendor-supported choice for interfacing your computer applications to PLCs and such things.

Which is not a problem by itself, since PLC, being an industrial equipment, should operate independently from a non-industrial equipment. The problem is idiots who think a desktop PC can reliably control PLC in real time.

Re: It's about what broke, not who broke it

#88
post #45
post #21

We have put a rule that made our system very strong over the years: We don’t care if you broke the site, just fix it quickly and more importantly write a test that will catch the same problem if it happens again. Every time someone breaks something, we get harder to break.

Sounds like your system is antifragile.

Robust. The word is "robust". We don't need to promote buzzwords.

Re: It's about what broke, not who broke it

#89

There is the risk of conflating two separate types of problem. There are problems that arise from the complexity of the code, and problems that arise from particular people. If a programmer has a habit of sloppy code, or violates the team's standards in some ways, then a good leader will keep track of the fact that one person is responsible for a recurring pattern of mistakes. I absolutely agree with Rachel By The Ba…

if the function was still there, I think it is also likely that the old jar or class file (with the function) was still lurking in the classpath or your version control and build system weren't using his revision

The point is, he failed to make any revisions. He was oddly disorganized. Even with quite a bit of coaching, he was unable to do what we needed.
Post reply on HN