Live data from Hacker News

The ‘flawed five’ engineering productivity metrics

leaddev.com

21–30 of 124 posts

Re: The ‘flawed five’ engineering productivity metrics

#21
post #9

There's some for SRE's and Sysadmins too: * Cost reduction; usually by a very arbitrary amount, despite you having no control over what's needed * Uptimes; last job told me that I had to get 99.998% uptime, Googles global load balancer is only 99.9%

You might as well just promise 100% uptime. If you don't meet it, most SLAs you're only liable for a couple bucks in service credits anyways.

I'm mostly talking about a metric by which my teams performance would be judged.

Externally to customers we had no promises of availability.

Re: The ‘flawed five’ engineering productivity metrics

#22
post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

These are great!

How are you using the last one? I feel like fraction of tasks started that reach customer could easily become misleading: you want to quickly abandon tasks once you realise they're no longer viable.

In fact, the development effort should be partially about finding reasons to stop working on the thing, so you can toss it out as soon as possible, instead of waiting for the customer to turn out not to use it.

I would even say that canceling many started tasks is directly correlated with a quick cycle time, by Little's law.

Re: The ‘flawed five’ engineering productivity metrics

#23
Decision making purely based on such metrics is wrong. It's management by numbers, and similarly like coloring by numbers, while relatively easy, will not produce great results.

At the same time, metrics do have a place. Even flawed metrics, like the ones this article describes can provide value. When used together with qualitative evaluation and thoughtful analysis, it provides a more complete picture of what is going on in a team.

Metrics such as these provide an addition perspective on a team. A manager knows what their team should be doing. A good manager should have an intuitive feel of what is going on. A manager should have a good qualitative idea of how their team is doing. If the metrics do not align with the other perspectives, something may be off.

If a manager believes a person should be coding, the person is not bringing up any challenges, is reporting progress, and they produced 3 small commits over the past month, it is time for a conversation to find out more about what is going on.

Re: The ‘flawed five’ engineering productivity metrics

#24
When leading these are the metrics you should care about:

Oldest MR - this should always be less than 2 weeks. This should normally be less than 1 week, but its not worth caring about at less than 2.

Unfinished sprints - sprints should finish with enough time left over to cope for an incident in the week. The extra time should be used for planning and continuous professional development (CPD). When someone is trapped in overflowing sprints it means they are deprioritising and undercompleting work that will come back to bite you.

Track other metrics for at most 3 months each, ideally only a month or a single week spot check. This prevents gaming and obsession whilst letting you reason about more nuanced behaviours.

Re: The ‘flawed five’ engineering productivity metrics

#26
post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

What do I care about start to finish time?

The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently.

A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress.

It's fundamentally not a synchronous process and I say a mistake to try to make it one.

Edit to address a sense I'm getting from several comments at once, that I may be a manager who doesn't understand or care about developers:

I AM the developer (& sysadmin) with a bunch of fresh and stale projects and it doesn't bother me at all. I always have work to do but it's not stressfull because that is about your boss(es) not about the number of open items. If anyone tried to say all that stuff needs to be done yesterday, sure that would be a problem. But all they really are is a spectrum of priorities. Some are just ideas little more than stubs for possible future interest, some would be nice but may never be justifiable in a strict easily quanifiable sense, you can't bill any customer for the hours, and yet would still be nice, would be one of the things that sets your product apart and attracts the customer in the first place, and some have normal priority, and when something high priority comes along, it simply displaces the rest. It just requires a boss who doesn't say that every single new item is always the most important.

That stack of available projects, and the age of half of them, doesn't bother me at all. Half the time, a different project is as good as a walk in the park for the reset / fresh look factor when stuck on something.

Re: The ‘flawed five’ engineering productivity metrics

#29
post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

I use a similar set:

- Time between idea first recorded and idea in production.

- Time between idea merged to main branch and idea in production (in some places this is negligible, in others where ideas spend most of their time hanging out -- done, but collecting dust).

- Time actively spent on task over the time between work started on idea and idea merged to main branch. (This is "flow efficiency" in lean terminology.)

Other than that I use the DORA metrics too. I measure defect rate by looking at proportion of merge requests that fix defects, because mostly that is a stable distribution.

Re: The ‘flawed five’ engineering productivity metrics

#30
In the past, I was a patent examiner at the USPTO. Patent examiners have their own problematic performance metrics. In Oct. 2020, the metrics went through a big change that made them a lot more complex. I suspect that part of the motivation was to make the system more opaque so that it'd be harder to game. But in practice I think it added just as many if not more ways to game the system.

I quickly figured out that under the new system, you could increase the amount of time you get for a particular patent application through a particular reclassification procedure called a C* (pronounced C-star) challenge. I'm surely not the only one who figured that out. The reason the C* challenge exists is to reclassify a patent so that it can be transferred to a more qualified examiner. But if it's not transferred then the amount of time you get can be changed. That's not necessarily nefarious as many applications have the wrong classification and would give you a lot less time than if they had the right classification. But examiners aren't incentivized to switch an application to the right classification. They're incentivized to change the classification so that the application gets transferred or change the classification so that they get more time. In the latter case I'd intentionally avoid adding (or even delete) any classifications that would reduce the amount of time I got. I don't suspect the long-term dynamics of this system are what USPTO management intends.

Post reply on HN