The ‘flawed five’ engineering productivity metrics • Hacker News Board

Re: The ‘flawed five’ engineering productivity metrics

#51

Post by Traubenfuchs » Mon, May 16, 2022, 5:45 PM UTC

foolfoolz wrote: post #32

velocity points is not useless. it can’t be used in isolation but points delivered by individuals is a great starting place to identify outliers in your org. generally if someone is delivering far higher or far fewer points they are making an outsized impact to the team (either positive or negative). it’s not perfect, you must take context with it, but with averages and on long time scales it’s quite reliable

They are COMPLETELY useless and are actively gamed by clever devs to reduce workload and reduce output expectations. I always nudge fellow devs to overestimate the tickets I will be working on by overstating the complexity and risks, sometimes I prime them with higher numbers, etc.

Re: The ‘flawed five’ engineering productivity metrics

#52

Post by Traubenfuchs » Mon, May 16, 2022, 5:48 PM UTC

xyzzy4747 wrote: post #4

The best metric is your own intuition about how productive people are being and their output, subjectively thinking about the quantity, quality, and impact. It becomes pretty obvious who is contributing a lot and who isn’t. You don’t need to track metrics.

This also doesn‘t work and just gives social, extroverted and eloquent people a huge advantage.

Re: The ‘flawed five’ engineering productivity metrics

#53

Post by dilyevsky » Mon, May 16, 2022, 5:56 PM UTC

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

So just rubberstamp your pr with a nit to score awesome on the first two?

Re: The ‘flawed five’ engineering productivity metrics

#54

Post by hatware » Mon, May 16, 2022, 6:00 PM UTC

dilyevsky wrote: post #53

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

So just rubberstamp your pr with a nit to score awesome on the first two?

You can game any system. I think the point is to give yourself guardrails that help you make good decisions for the present and future.

Re: The ‘flawed five’ engineering productivity metrics

#55

Post by feoren » Mon, May 16, 2022, 6:03 PM UTC

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

> - Avg time to complete code review

Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.

Re: The ‘flawed five’ engineering productivity metrics

#56

Post by yodon » Mon, May 16, 2022, 6:04 PM UTC

dilyevsky wrote: post #53

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

So just rubberstamp your pr with a nit to score awesome on the first two?

> So just rubberstamp your pr with a nit to score awesome on the first two?

If you're saying you can CR your own commits, that's a different problem to discuss unrelated to the numbers.

If you're saying you're coordinating so closely with the other devs on your team that they're picking up your CR's immediately and they're able to legitimately approve them with just a trivial nit, then congrats - you're "gaming" the system in exactly the way it's designed to be "gamed." In general, if devs are talking to each other and aware of what each other is doing, efficiencies are high and latencies are short. If no one is talking to each other, efficiencies are low and latencies are high. Talking to each other and coordinating with each other is a good thing.

Re: The ‘flawed five’ engineering productivity metrics

#57

Post by yodon » Mon, May 16, 2022, 6:14 PM UTC

feoren wrote: post #55

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

> - Avg time to complete code review Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.

> Doesn't [Code Review time] depend heavily on the size of each PR?

The amount of active "eyeballs on code" time to complete a code review can of course vary tremendously. That said, when I see code reviews taking a week to complete, only very very very rarely is that a sign that developers are routinely spending a solid week of eyeballs-on-code time completing those code reviews. Most of the time it involves a relatively small amount of focused review time and a huge amount of not actually getting around to starting doing the code review time.

If everything in your shop is so well handled that you're not wasting any time waiting for code to get reviewed, in practice your shop is probably also already making good choices about how to structure code for efficient reviews as well (but that's more an anecdotal observation about correlations, not directly an observation about causality).

Re: The ‘flawed five’ engineering productivity metrics

#58

Post by vannevar » Mon, May 16, 2022, 6:19 PM UTC

I think the real message here is not that metrics are bad, but that they are misused. Imagine if every time you went to the doctor with a fever and they took your temperature, the doctor prescribed an ice bath to bring your temperature down. You wouldn't conclude that thermometers are evil, you'd switch doctors. Same goes for most of the metrics here.

Metrics are useful to navigate BY, not to navigate TO. If you have skilled and experienced managers, you can get a lot of value out of all of the metrics listed in the article.

Re: The ‘flawed five’ engineering productivity metrics

#59

Post by ethanwillis » Mon, May 16, 2022, 6:22 PM UTC

ok123456 wrote: post #18

shakezula wrote: post #13

For fuck’s sake, can we just stop trying to measure developer productivity like we’re an assembly line?

But then how will non-technical managers justify their salary?

I know the question is probably sarcastic. But being serious: Simply by being good at their jobs of helping the people they manage and having those people want them to be part of the team. And then on the flipside as well where higher ups trust them to help be a good translation layer that enables teams to meet organizational objectives.

Re: The ‘flawed five’ engineering productivity metrics

#60

Post by kodah » Mon, May 16, 2022, 6:22 PM UTC

prepend wrote: post #41

kodah wrote: post #17

The problem with all of these metrics is that they assume that there's an engineering team, or someone on an engineering team, somewhere that's not doing shit . Whether the business believes they're not doing shit because they refactor more than they write features, because they release twice a year instead of every week, or because they have less frequent merges. The entire emphasis of measurement is squarely a tech…

I’ve worked with teams and individuals that do nothing. Like literally nothing, they do shit. I once had to wrap up a product release for contract close out or something and it involved getting all the code and commits from developers who were rolling off. It amazed me how many had zero code they had written in months. I had one developer that had never committed anything in the three months he was there. Obviously t…

I believe you, I'm sure they exist, my argument is that they're not common enough for this level of pain. Personally, the way that org is sounds like it was by design.