Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's about what broke, not who broke it

rachelbythebay.com

Post reply on HN
Print view View original

11–20 of 105 posts

Re: It's about what broke, not who broke it

#11

Post by smallbigfish »

nevatiaritika wrote: post #7

My manager at work especially has the reverse attitude where the person who broke it is more significant than what broke/how we fixed it/ how to avoid it in the future. I have seen people get taunted for a bug they caused two years ago, a bug which didn't affect any revenue or was pretty easy to fix. And of course it still gets pointed out during appraisals. Its a nightmare, because there's no room for experiment lef…

We don't touch production. We don't upgrade. We are are a X million company we can't afford the risks.

These are some of the excuses they put up.

And then they sit 10 years or more with that bad stuff in there, build even uglier ways around it.

But the time comes to actually do something about. And what was once a one day job becomes "we will hire a consultancy firm to guide us".

Re: It's about what broke, not who broke it

#12

Post by voltagex_ »

BrissyCoder wrote: post #4

I don't know. Where I work no discernible pattern can be found with the "what" that broke. It's always the same f *ing people that break it though!

If I had a developer who was breaking stuff often enough to worry:

* Do they have too much access to systems?

* Is there something really wrong with the deployment system?

* What training can be provided?

All of that is more constructive than your comment, as cathartic as it may be.

Re: It's about what broke, not who broke it

#13

Post by iamdave »

smallbigfish wrote: post #11
nevatiaritika wrote: post #7

My manager at work especially has the reverse attitude where the person who broke it is more significant than what broke/how we fixed it/ how to avoid it in the future. I have seen people get taunted for a bug they caused two years ago, a bug which didn't affect any revenue or was pretty easy to fix. And of course it still gets pointed out during appraisals. Its a nightmare, because there's no room for experiment lef…

We don't touch production. We don't upgrade. We are are a X million company we can't afford the risks. These are some of the excuses they put up. And then they sit 10 years or more with that bad stuff in there, build even uglier ways around it. But the time comes to actually do something about. And what was once a one day job becomes "we will hire a consultancy firm to guide us".

Are you my coworker? Because you sound like him /s

No really I appreciate risk management but when it cripples your ability to make decisions, innovate or otherwise ACT on information that could help you be a more efficient team and the develoment team becomes a room full of people doing nothing but maintenance for years on end people leave and companies fail.

I just watched that very thing happen this year to my company for exactly that reason. Someone with the word "senior" in their job title was so risk averse that the market caught up, passed us and started eating our lunch.

God help them because I can't do it anymore, and writing on the wall says they'll be closing up shop this fall. I'm out the door for good at the end of the week.

Re: It's about what broke, not who broke it

#14

Post by tzhenghao »

dancek wrote: post #10

I think this is an important piece of organization culture. If the first reaction to problems is blame and punishment, issues are covered up. But if finding bugs and fixing them is considered valuable, there will be less issues in the long run. Of course I write enough stupid bugs myself that I'm bound to think this way.

This is so true. Providing the incentive to squash bugs than punishing people for making them is the driving force for innovation in a team. Take that away, and you get a toxic culture where everybody starts finger pointing when an issue arises.

Re: It's about what broke, not who broke it

#15

Post by badminton1 »

If in soccer the opposing team scores, who is to blame? the goalkeeper, defenses? the coach? the whole team? the referee? nobody?

Preventing goals means that the strategy needs to ensure good ball possession, and staying on the offense, to reduce the burden on the defense, to reduce the burden on the goalkeeper, who is the last line of defense.

If the last line of defense fails that's not an individual failure but a team failure, coach included, since the coach selects who gets to play, when and their roles.

Same in software: bad management passes the burden to developers, bad development passes the burden to testers, bad testing passes the burden to release management.

Re: It's about what broke, not who broke it

#16

Post by krallja »

nstj wrote: post #6

I like this site and hadn't really read much from it - it's interesting how much it's been front paged over the last couple of weeks: https://news.ycombinator.com/from?site=rachelbythebay.com

Rachel is an excellent writer who was on a long break from writing. Seems like HN is happy to read her posts again.

Re: It's about what broke, not who broke it

#17

Post by altano »

CoolGuySteve wrote: post #3

I used to think this way until I started working with someone who was nearly always the one who broke it. At some point we just had to face the fact that his work was unreliable even after significant mentoring. If the tasks were difficult that would be one thing, but I'm talking about stuff like committing code to prod that was clearly never even executed once.

Sounds like you have a code review and automated testing problem and not a bad coworker problem.

Re: It's about what broke, not who broke it

#18

Post by lawn »

altano wrote: post #17
CoolGuySteve wrote: post #3

I used to think this way until I started working with someone who was nearly always the one who broke it. At some point we just had to face the fact that his work was unreliable even after significant mentoring. If the tasks were difficult that would be one thing, but I'm talking about stuff like committing code to prod that was clearly never even executed once.

Sounds like you have a code review and automated testing problem and not a bad coworker problem.

Sounds like both.

Re: It's about what broke, not who broke it

#19

Post by saagarjha »

BrissyCoder wrote: post #4

I don't know. Where I work no discernible pattern can be found with the "what" that broke. It's always the same f *ing people that break it though!

Is there a reason why it's breaking, though? Is it really because the person breaking it is incompetent, or is it because there wasn't enough documentation or education or safeguards in place to prevent this from happening?

Re: It's about what broke, not who broke it

#20

Post by csours »

I took down an assembly plant by clicking on a Network status icon from a particular hardware supplier.

Over the weekend, firmware patches were applied, and the server rebooted. After reboot, everything worked fine, so the tech marked the change successful and went home.

Well, apparently the NICs would work just fine, but not all settings were applied until you opened the UI provided by the vendor. When you opened the UI, the final settings would be applied, and the NICs would reboot, just long enough to kill TCP connections.

That loss of TCP connection killed the parent system, and then all the other children systems also died when the parent died.

So who would you even blame there? The guy who set the tripwire? The guy who tripped on the tripwire? The guy who designed a system that could be brought down by a momentary loss of connection?

I'm lucky that my boss wasn't the type to point fingers, because I was the guy who was there when it happened, and it sure got a lot of attention.

Post reply on HN