Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The ‘flawed five’ engineering productivity metrics

leaddev.com

Post reply on HN
Print view View original

81–90 of 124 posts

Re: The ‘flawed five’ engineering productivity metrics

#81

Post by yodon »

dilyevsky wrote: post #76
yodon wrote: post #56

Earlier quoted context omitted.

> So just rubberstamp your pr with a nit to score awesome on the first two? If you're saying you can CR your own commits, that's a different problem to discuss unrelated to the numbers. If you're saying you're coordinating so closely with the other devs on your team that they're picking up your CR's immediately and they're able to legitimately approve them with just a trivial nit, then congrats - you're "gaming" the…

So you’re becoming “aware of what each other is doing” by half reading the pr title and going straight to approve button? I think “legitimately “ is doing a lot of lifting here too

You seem to presuppose that the people involved in the team conversations about these metrics are stupid or clueless to a point of willfulness or intentionally bad actors. If that's the case, as with the case where you're somehow able to approve your own CR's, you face problems unrelated to the numbers. If those are the kind of problems you face, you need to fix them well before you try to bring in any processes designed to help a well functioning organization function better, including the case you called out where devs are approving code reviews without understanding them.

Re: The ‘flawed five’ engineering productivity metrics

#82

Post by closedloop129 »

shakezula wrote: post #47
brazzy wrote: post #25

Earlier quoted context omitted.

So how do we measure it?

Why do we need to? No, seriously, why? I have yet to see any meaningful increase in a team’s productivity after they start tracking “developer productivity”. Each time it results in a blow to developer morale and a pretty dashboard that management uses to retroactively justify their decisions.

Have they just started tracking “developer productivity” or do they also spend time on identifying good developers and enabling them to teach their skills?

If you want some form of rational decision when you develop a team, how else but with metrics do you make decisions?

Re: The ‘flawed five’ engineering productivity metrics

#83

Post by dkersten »

kqr wrote: post #62
feoren wrote: post #55

Earlier quoted context omitted.

> - Avg time to complete code review Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.

Have you measured this in your organisation? In my experience, every team has a fairly distinct cut-off point where if you make PRs larger than that, they start to hang around in someone's review queue forever. Every time I've made a larger PR I've regretted it and subsequently split it up into smaller ones. I get higher quality feedback and faster feedback to boot that way.

> if you make PRs larger than that, they start to hang around in someone's review queue forever.

Or you just get a "ship it" without more than a cursory glance.

Re: The ‘flawed five’ engineering productivity metrics

#84

Post by marcosdumay »

lukeramsden wrote: post #79
marcosdumay wrote: post #71

Earlier quoted context omitted.

> Fraction of tasks started that never reach a customer If you optimize this down, you punish any kind of innovative or ambitious task.

Depends how you measure "reaching", i.e. all refactorings and optimisations all reach the customer technically "reach" the customer as soon as they're deployed (essentially, anything that meets its definition-of-done is "delivered). I think the wider idea is to measure work being interrupted and abandoned, or being abandoned due to not being necessary, both of which could be dubbed signs of inefficiency.

Hum, no. I'm talking exactly about abandoned work.

If you don't abandon work, you either don't take any risky task, or you create a really shitty environment and will lose effectiveness soon because of it anyway.

And yes, for some software you don't actually need to take any risky task. At least for a while. But if you are in a position to set that policy, you are very likely not in a position to have a clear enough view to know if your software is such one.

Re: The ‘flawed five’ engineering productivity metrics

#85

Post by the_jeremy »

lordnacho wrote: post #40
Brian_K_White wrote: post #26

Earlier quoted context omitted.

What do I care about start to finish time? The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently. A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress. It's fundamentally not a synchronous…

You need idle time as an engineer. Otherwise if a higher value item appears in the pipeline after a lower value item you'll be working on the lower value thing first. Idle time is also where you do useful things like checking for updated dependencies, running a linter, generally reading up on the tech you're using, and making the code ready for future edits. Not so idle at all but not really the same as working on a…

Do you work somewhere that gives you this idle time? We explicitly have tickets for any tech debt or dependency updates we need to do, because otherwise we don't get time to do that.

Re: The ‘flawed five’ engineering productivity metrics

#86

Post by yodon »

marcosdumay wrote: post #84
lukeramsden wrote: post #79

Earlier quoted context omitted.

Depends how you measure "reaching", i.e. all refactorings and optimisations all reach the customer technically "reach" the customer as soon as they're deployed (essentially, anything that meets its definition-of-done is "delivered). I think the wider idea is to measure work being interrupted and abandoned, or being abandoned due to not being necessary, both of which could be dubbed signs of inefficiency.

Hum, no. I'm talking exactly about abandoned work. If you don't abandon work, you either don't take any risky task, or you create a really shitty environment and will lose effectiveness soon because of it anyway. And yes, for some software you don't actually need to take any risky task. At least for a while. But if you are in a position to set that policy, you are very likely not in a position to have a clear enough…

In practice I generally see the abandoned work fraction dominated by much less interesting and much less valuable poor planning decisions, false starts, and misalignment between teams. Genuine high-value risk taking is typically already so rare in most organizations that it doesn't even show up significantly in a count like this. Equally importantly, when those genuine risk taking activities are terminated they typically happen in the form of an experiment that reached a customer at some level as part of the decision to terminate the effort. If it reached a customer in some manner like that it's not in the bucket of tasks started that never reached a customer.

Re: The ‘flawed five’ engineering productivity metrics

#87

Post by owlbite »

Same old story - as soon as you start using a metric to incentivize people, they optimize to the metric. If the metric is not well aligned with what you actually wanted (and I mean optimizing it is what you want, not currently correlated before you incentivized it), you are not going to be happy.

It always amazes me how otherwise very smart people don't think through the consequences of "paid by the X".

Re: The ‘flawed five’ engineering productivity metrics

#88

Post by imoverclocked »

feoren wrote: post #55
yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

> - Avg time to complete code review Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.

Just starting the review can take a while. However, engineers who might be reviewing code instead of other activities may be providing more value/higher velocity/better quality with those other tasks than breaking to just review a PR.

Someone once argued to me that each feature PR should have very few surprises if the feature has been planned and communicated well with affected parties before implementation even begins. The opposite is often true in some organizations.

Re: The ‘flawed five’ engineering productivity metrics

#89

Post by rightbyte »

madcaptenor wrote: post #33

My kid is figuring out pooping in the potty. We have a chart where we make a check mark when she does it. She likes check marks, especially when she can make them herself. Over the past couple weeks her poops have gotten smaller and more frequent.

You should make bigger marks for bigger deliveries to not give the wrong incentives.

Re: The ‘flawed five’ engineering productivity metrics

#90

Post by gary_0 »

"When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law. See also, the Cobra Effect: https://en.wikipedia.org/wiki/Perverse_incentive.

If you set up an incentive system, people will (perhaps not even consciously) start trying to game that system. Maybe there's some way to use machine learning to turn performance metrics into a black box that considers every possible data point? (Now there's a terrifying idea.)

Post reply on HN