The ‘flawed five’ engineering productivity metrics • Hacker News Board

Re: The ‘flawed five’ engineering productivity metrics

#31

Post by wnolens » Mon, May 16, 2022, 5:14 PM UTC

ptudan wrote: post #6

Amazon promos and firings are based a lot around the amount of lines of code you write, the number of code reviews you do (and the percentage of the time you review when asked), the number of merge requests you have, and the number of iterations per review. If you average more than 2 iterations per MR, you're on the chopping block as it means you're "sloppy". Its ridiculously dumb. I've heard those numbers matter les…

Are you a manager at Amazon employing these or seeing your peers employ these metrics?

I've never heard of such a thing. It's leadership principles only AFAIK.

Re: The ‘flawed five’ engineering productivity metrics

#32

Post by foolfoolz » Mon, May 16, 2022, 5:16 PM UTC

velocity points is not useless. it can’t be used in isolation but points delivered by individuals is a great starting place to identify outliers in your org. generally if someone is delivering far higher or far fewer points they are making an outsized impact to the team (either positive or negative). it’s not perfect, you must take context with it, but with averages and on long time scales it’s quite reliable

Re: The ‘flawed five’ engineering productivity metrics

#33

Post by madcaptenor » Mon, May 16, 2022, 5:18 PM UTC

My kid is figuring out pooping in the potty. We have a chart where we make a check mark when she does it. She likes check marks, especially when she can make them herself. Over the past couple weeks her poops have gotten smaller and more frequent.

Re: The ‘flawed five’ engineering productivity metrics

#34

Post by kqr » Mon, May 16, 2022, 5:21 PM UTC

Brian_K_White wrote: post #26

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

What do I care about start to finish time? The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently. A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress. It's fundamentally not a synchronous…

This is a common conflict!

One perspective says that idle engineers cost money, so you should load up your engineers to 99 % utilisation with a huge backlog of tasks. The consequence of this is that tasks take, on average, months to complete because they sit idle most of the time, not being worked on.

The other perspective is that idle tasks is what's expensive. Thus you must keep your engineers lightly loaded (say 70 %) so they can, in your words, pounce on new tasks. With this setup, most tasks spend a negligible time waiting and most time actively worked on.

Now, how can idle tasks be more expensive than idle engineers? There are whole books dedicated to explaining this. I'd recommend starting with Reinertsen's Principles of Product Development Flow.

Re: The ‘flawed five’ engineering productivity metrics

#35

Post by dcow » Mon, May 16, 2022, 5:22 PM UTC

Brian_K_White wrote: post #26

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

What do I care about start to finish time? The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently. A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress. It's fundamentally not a synchronous…

Some developers are very much not efficient in a "fully pipelined" stage. Lots of context switching can slow down everything in the pipeline. Even if you're waiting on a code review and burn a few hours, it can still be more efficient to stay focused on driving the task at hand to completion than trying to spin up a small task during that time and switching back to respond to review comments every so often.

Re: The ‘flawed five’ engineering productivity metrics

#36

Post by ptudan » Mon, May 16, 2022, 5:23 PM UTC

wnolens wrote: post #31

ptudan wrote: post #6

Amazon promos and firings are based a lot around the amount of lines of code you write, the number of code reviews you do (and the percentage of the time you review when asked), the number of merge requests you have, and the number of iterations per review. If you average more than 2 iterations per MR, you're on the chopping block as it means you're "sloppy". Its ridiculously dumb. I've heard those numbers matter les…

Are you a manager at Amazon employing these or seeing your peers employ these metrics? I've never heard of such a thing. It's leadership principles only AFAIK.

I worked there for a bit over 2 years, pretty recently. I heard this from other engineers and from managers not directly in my chain.

The iterations per MR metric is one I know for a fact is used in PIP docs. Heard this directly from a friend that was PIPed.

Re: The ‘flawed five’ engineering productivity metrics

#37

Post by prepend » Mon, May 16, 2022, 5:24 PM UTC

Does anyone actually use these as continuous variables and evaluate them. I’ve worked for 10 orgs for almost 30 years and while these existed, I’ve never even heard someone propose to use them to measure productivity.

#commits are useful as a binary metric that a developer is alive, but trying to say one is more productive than another because they had more commits is pure madness that permeates an org so that I would detect it during an interview and avoid.

Re: The ‘flawed five’ engineering productivity metrics

#38

Post by yodon » Mon, May 16, 2022, 5:26 PM UTC

kqr wrote: post #22

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

These are great! How are you using the last one? I feel like fraction of tasks started that reach customer could easily become misleading: you want to quickly abandon tasks once you realise they're no longer viable. In fact, the development effort should be partially about finding reasons to stop working on the thing, so you can toss it out as soon as possible, instead of waiting for the customer to turn out not to u…

> Why are you using the last one?

The last one is definitely in a different bucket for me than the first four. For starters, all of these are there primarily to encourage conversation. That said, the first four can make a lot more sense to try to graph and track and optimize. The last one tends to be more purely about driving a conversation.

At the top level, if you're genuinely doing a ton of learning along the way to shipping the right feature to the customer, then arguably that value is ultimately reaching the customer. If on the other hand you can't decide what the goal is and you keep changing your mind (as is often the case), then you tend to end up with a lot of dev investment made in things that simply never ship.

It's also worth differentiating technical "spikes" from feature "experiments." In my vocabulary, spikes are things where you're internally assessing a question like "could we do this" and experiments are things where you're externally assessing "do customers want this/does this have the impact we want." If you have a lot of experiments that don't reach customers, you're burning a lot of dev time on things that aren't actually experiments (because by this definition experiments need to reach the customer surface to deliver data). That's a signal you should probably be looking at. Spikes generally only reach the customer indirectly (through an eventual shipping feature), but if you have a lot of spikes that don't ever reach the customer in any way that's also a signal you should probably be looking at.

Re: The ‘flawed five’ engineering productivity metrics

#39

Post by kqr » Mon, May 16, 2022, 5:26 PM UTC

koliber wrote: post #23

Decision making purely based on such metrics is wrong. It's management by numbers, and similarly like coloring by numbers, while relatively easy, will not produce great results. At the same time, metrics do have a place. Even flawed metrics, like the ones this article describes can provide value. When used together with qualitative evaluation and thoughtful analysis, it provides a more complete picture of what is goi…

Yup. Performance measurement is like planning: the outcome (metrics, a plan) is useless. The process you take to getting there (discovering, questioning, measuring, imagining, simulating) is everything.

Re: The ‘flawed five’ engineering productivity metrics

#40

Post by lordnacho » Mon, May 16, 2022, 5:27 PM UTC

Brian_K_White wrote: post #26

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

What do I care about start to finish time? The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently. A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress. It's fundamentally not a synchronous…

You need idle time as an engineer. Otherwise if a higher value item appears in the pipeline after a lower value item you'll be working on the lower value thing first.

Idle time is also where you do useful things like checking for updated dependencies, running a linter, generally reading up on the tech you're using, and making the code ready for future edits. Not so idle at all but not really the same as working on a ticket either.