Live data from Hacker News

Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

web.mit.edu

331–340 of 446 posts

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#331
post #144

Went through this exact scenario at my job recently. As the tech lead/architect of an org I reviewed a bunch of recently released projects and identified areas that needed critical improvements because they had major reliability/performance issues. One team had multiple releases at the top of the list, and their PM and eng manager (and really everyone else up the chain) simply ignored all concerns because they had to…

Well, you could dust off the old email and 'mistakenly' send it again. Though it will seem petty, a small fraction might rethink the past few months.

I’ve started doing something like this more frequently, but it is in the interest of job security / cover-my-butt rather than pettiness.

I build something that depends on many systems that others have created and that uses data from probably about every dataset we have. What I build also has very high customer visibility. So whenever something breaks upstream, the first place customers notice the problem is in the system I maintain. Psychologically, people begin to develop a mental association around your system being a “problem” if it keeps getting brought up as the starting point of SEV discussions and customer tickets.

As a result, a lot of what I do now is defensive. I investigate and reverse engineer upstream codebases to identify likely failure modes, and I spend hours analyzing datasets of questionable origin for data quality issues and inconsistencies. I document and date stamp all of the problems that I find, file bug reports and assign them to the relevant teams, and write proposals describing potential solutions to what I see as large architectural design flaws that will come back to haunt us at some point.

All of this work is promptly ignored with “not enough bandwidth right now” or “not a priority compared to feature development”. Which is fine. I document all of those responses too.

Then eventually, something breaks in a big way that is again first noticed by customers within the system I am responsible for. In the past, I would immediately drop what I was doing and scramble into investigation mode for a few days to prove I wasn’t the root cause of the X hundred thousand dollar issue, er... I mean the blameless SEV review & postmortem... but more recently my preemption seems to be paying off, and lately I just reply to the panic with a bunch of links to old Slack threads (where we already discussed the issue), the documents and proposals I created (that no one read), or the bug reports I filed (that were never followed up on).

Perhaps it does come across as a bit petty, but I try to be as polite as I can, and from my perspective it’s an improvement over the previous situation. The only downside is that all of this preventative work takes away time from the primary work that I was hired to do.

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#332
post #304
post #272

Earlier quoted context omitted.

Great workers know when they can help others without falling behind themselves. Or have good communication skills to explain what they were doing.

Or are good at communicating and don't need to do anything because they can just lie.

Well at some point you expect some artifacts to show up, even if not just code, design docs, RFCs, code reviews, something. If there truly is nothing I don't think that's a viable way to contribute either. Sure one can "only help others" but all good engineers I've worked with can help others and still do their main tasks, and if they get to a point where they are a point of reference for everyone else and everyone else gets blocked without them this is a high priority thing to fix in the team, probably one of the highest priority things to fix.

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#333

Earlier quoted context omitted.

I don't think it can't be done right, but it's been cargo-culted pretty hard at Fortune 500 companies. But the best scrums I have ever done have just been literally post-it notes on a whiteboard - I think Jira ads a level of complexity that is unsustainable for most orgs.

In my 17 years in the industry I saw working agile twice. Both were small places, both used post-it notes. Both were consultancies, each team dedicated to an independent project. Everything else, world-class, public, private, consultants or not... was a joke. I don't think cross-team collaboration works in agile. Either one plans ahead, or everything becomes an unpredictable mess.

Similar, but one place, and same with post-it notes. It went down the shitter when Jira was force fed and all the problems related to keeping them in sync arose.

"b-b-but other teams need visibility of your sprint!" no they don't. I don't think I've looked at other board other than by accident.

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#334
post #332
post #304

Earlier quoted context omitted.

Or are good at communicating and don't need to do anything because they can just lie.

Well at some point you expect some artifacts to show up, even if not just code, design docs, RFCs, code reviews, something. If there truly is nothing I don't think that's a viable way to contribute either. Sure one can "only help others" but all good engineers I've worked with can help others and still do their main tasks, and if they get to a point where they are a point of reference for everyone else and everyone e…

I've seen management let bad engineers re-do projects from scratch 2 or 3 times. Because they blame bad requirements, poor choice of programming language, and whatever on all the issues that the project has.

They eventually acquire enough experience to be able to produce something that sort-of works around the 3rd attempt.

But thanks to how good they are at communicating, are considered by management to be good engineers.

While doing a working project at the 1st try, without using this week's new framework and so on isn't as valued.

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#335
post #27

I've been contemplating the issue in the headline, as pertains to my value. When I help somebody in 40 mins with something they've been stuck on for 3 months, my value is clear to everyone. When I work there the whole time, and nobody ever gets stuck for 3 months, my value is unclear. Don't know how to deal with this paradox.

I am dealing with similar. I can provide solutions 10x faster than anyone. Everyone looks up to me, and are dependent on me for solutions to everything.

However it is clear that noone is improving and that the process IS me.

What I've been trying to do is make sure that my scope and role is fully clarified, and any "extra" activity that I perform is documented and flagged. Anything that becomes a "common" activity implies a missing part of the process - be it a role that is missing, or a skillset that is lacking.

Before you start thinking that it is pretentious or self serving, it's perhaps the opposite - you owe it to the process and the team/business/organisation to make them see what you're putting in, else they fail to find the gap.

It's not paid off yet, but hopefully it will yield results within the next 6 months.

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#336

This us an incentives problem, plain and simple. Its extremely difficult to quantify things that don't happen, and because we've been building our economies and societies around a heavy focus on quantitative metrics we end up only incentivizing people to solve problems. Even create problems on purpose if you need to, you'll get credit for fixing them. Human Resources departments often run into this exact scenario. Ma…

I wonder if this is why some managers seem to go out of their way to generate chaos. I have no patience for it, and judge them harshly for it. They don’t get any credit if my book for “solving” problems they pulled out of their back side or willfully ignored planning for.

I'm sure there are some that do this.

I once worked with a test manager that was effectively judged by how many bugs were found and how many tests were written. He was actually a nice guy just trying to play the game given to him, but if you didn't know this you would think he was doing exactly as you described.

There will always be a few bad eggs in the mix, but I'd check the incentives forced on them before judging too harshly.

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#337
post #334
post #332

Earlier quoted context omitted.

Well at some point you expect some artifacts to show up, even if not just code, design docs, RFCs, code reviews, something. If there truly is nothing I don't think that's a viable way to contribute either. Sure one can "only help others" but all good engineers I've worked with can help others and still do their main tasks, and if they get to a point where they are a point of reference for everyone else and everyone e…

I've seen management let bad engineers re-do projects from scratch 2 or 3 times. Because they blame bad requirements, poor choice of programming language, and whatever on all the issues that the project has. They eventually acquire enough experience to be able to produce something that sort-of works around the 3rd attempt. But thanks to how good they are at communicating, are considered by management to be good engin…

I mean at some point we just get generic enough that we have to agree on the ultimate fact that at least somewhere in the management chain someone has to actually know enough engineering to recognize good work from just running around in circles. I agree what you say happens but if good work isn't consistently recognized and you think you've worked on your comms, there's not much else to do really.

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#338
post #55

Earlier quoted context omitted.

I'm sure I'm not the only one that can't wait for the 2038 problem to get closer. bank accounts gonna be jumping like a 3-peat Micheal Jordan when those consultation fee checks start rolling in.. provided the Boston Dynamics dogs haven't taken over yet

Java EE will be the COBOL of 2037...

Don't insult COBOL like that

Re: Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

#340
post #8
post #3

In about that same timeframe, Y2K is also really good example. Nothing of note really happened. But if people had just ignored it a lot of things probably would have.

This is a very important case. There was an enormous investment in fixing Y2K, starting long before the day in question (e.g. financial instruments with expiration dates after Y2K had to be fixed before they became due), so on the day there was only a smattering of minor residual bugs. There were a few jokes about it in the newspapers but by and large the public just ignored it. I work on climate and hope / hoped tha…

Since you "work on climate" I am sure it got your attention that there is a massive effort happening all across the board. R&D has made tremendous advances, whole industries are investing to get away from fossil fuels, and governments have ongoing programs to accelerate and support lange hydro, solar, and wind projects.

Some countries have not heard the call yet it seems (ones that like their role as perpetual victims and blame everyone else but themselves), but even just five years down the road, the world will already look very different for all the effort that's being made.

Post reply on HN