The ‘flawed five’ engineering productivity metrics • Hacker News Board

Re: The ‘flawed five’ engineering productivity metrics

#41

Post by prepend » Mon, May 16, 2022, 5:28 PM UTC

kodah wrote: post #17

The problem with all of these metrics is that they assume that there's an engineering team, or someone on an engineering team, somewhere that's not doing shit . Whether the business believes they're not doing shit because they refactor more than they write features, because they release twice a year instead of every week, or because they have less frequent merges. The entire emphasis of measurement is squarely a tech…

I’ve worked with teams and individuals that do nothing. Like literally nothing, they do shit. I once had to wrap up a product release for contract close out or something and it involved getting all the code and commits from developers who were rolling off. It amazed me how many had zero code they had written in months. I had one developer that had never committed anything in the three months he was there.

Obviously this is a management problem and it’s not a single individuals fault. But the manager had like 90 contractors reporting to them and didn’t care that people had zero lines of code written.

The developer’s job was to code. I wanted to mention that as there are developers and designers who don’t code but are productive in other ways.

Re: The ‘flawed five’ engineering productivity metrics

#42

Post by dandare » Mon, May 16, 2022, 5:29 PM UTC

One thing that baffles me in corporate IT is not just the snake pace of development but rather the fact that nobody seems to be bothered by the snake pace. There is zero effort to measure or speed things up. The only important thing is to be nice to everyone, any mention of productivity is considered hostile behaviour.

(For reference, I am talking about cases where a team of 5 devs takes 2-3 months to deliver a feature that would take a single independent developer maybe 2-3 days.)

Re: The ‘flawed five’ engineering productivity metrics

#43

Post by prepend » Mon, May 16, 2022, 5:30 PM UTC

foolfoolz wrote: post #32

velocity points is not useless. it can’t be used in isolation but points delivered by individuals is a great starting place to identify outliers in your org. generally if someone is delivering far higher or far fewer points they are making an outsized impact to the team (either positive or negative). it’s not perfect, you must take context with it, but with averages and on long time scales it’s quite reliable

I think velocity points are useful within a team over time. They are locally useful.

But they are stupid to measure across teams or to compare teams or productivity. Velocity points are just an estimating tool, not a measure of value. It’s useful to know that a team usually produces 10 points per sprint but this sprint is 5 or 20. It just lets you know if your team is producing “normal” or not.

It’s useless to try to calculate that out of 20 teams the average velocity points are 10 per sprint.

Re: The ‘flawed five’ engineering productivity metrics

#44

Post by district487 » Mon, May 16, 2022, 5:30 PM UTC

kqr wrote: post #22

yodon wrote: post #12

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics…

These are great! How are you using the last one? I feel like fraction of tasks started that reach customer could easily become misleading: you want to quickly abandon tasks once you realise they're no longer viable. In fact, the development effort should be partially about finding reasons to stop working on the thing, so you can toss it out as soon as possible, instead of waiting for the customer to turn out not to u…

not the OP, but I would use it as a leading indicator that there is something wrong with the way we're scoping and prioritizing tasks. If 100% of our tasks are no longer viable, it's a signal that we're doing something fundamentally wrong earlier in the chain, e.g. are we misunderstanding the user problems? the market? the wrong prioritization? not enough scoping done?

Re: The ‘flawed five’ engineering productivity metrics

#45

Post by teeray » Mon, May 16, 2022, 5:31 PM UTC

I’ll add another one: code coverage. Coverage is, at best, a proxy metric for how easy it is to test your code. If you make it easy to write tests, coverage generally takes care of itself.

Re: The ‘flawed five’ engineering productivity metrics

#46

Post by prepend » Mon, May 16, 2022, 5:33 PM UTC

dtagames wrote: post #2

The most useful metric in my view, and one I learned at IBM, is fixes applied over shipped lines of code. Multiple fixes over the same lines of code is exponentially bad. IBM started measuring code defects vs working code in this way because productivity studies they did showed that fixes took much more time per LOC than new code and had other costs (customer sat, doc changes, reputation) besides.

This seems easily gamed by just producing lots of lines of code.

Adding inline documentation would improve this metric.

This does explain some absolute dogshit products IBM made as maybe they were optimizing for this metric by having 1000 lines when one would do. I’m bitter from having to decompile and debug websphere in the 90s and 00s.

I think this runs into a problem is that programmers are good at minmaxing. So any rote metrics will end up being gamed pretty quickly.

Re: The ‘flawed five’ engineering productivity metrics

#47

Post by shakezula » Mon, May 16, 2022, 5:40 PM UTC

brazzy wrote: post #25

shakezula wrote: post #13

For fuck’s sake, can we just stop trying to measure developer productivity like we’re an assembly line?

So how do we measure it?

Why do we need to? No, seriously, why?

I have yet to see any meaningful increase in a team’s productivity after they start tracking “developer productivity”.

Each time it results in a blow to developer morale and a pretty dashboard that management uses to retroactively justify their decisions.

Re: The ‘flawed five’ engineering productivity metrics

#48

Post by Traubenfuchs » Mon, May 16, 2022, 5:41 PM UTC

dandare wrote: post #42

One thing that baffles me in corporate IT is not just the snake pace of development but rather the fact that nobody seems to be bothered by the snake pace. There is zero effort to measure or speed things up. The only important thing is to be nice to everyone, any mention of productivity is considered hostile behaviour. (For reference, I am talking about cases where a team of 5 devs takes 2-3 months to deliver a featu…

A slow pace makes it possible to slack off more in peace. If you plan to take a week to do feature A and you finish it in a day, you have 4 free days. If you plan for 1 day and it takes 2 because it was harder than expected, plans get messed up, you need to work faster on the next feature and look bad.

I always encourage fellow engineers to vastly overestimate tickets. That‘s also important to set a comfortable pace with the business people who have zero clue how hard our work really is and prevent them from making us work hard.

Re: The ‘flawed five’ engineering productivity metrics

#49

Post by buscoquadnary » Mon, May 16, 2022, 5:43 PM UTC

jerf wrote: post #16

ptudan wrote: post #6

Amazon promos and firings are based a lot around the amount of lines of code you write, the number of code reviews you do (and the percentage of the time you review when asked), the number of merge requests you have, and the number of iterations per review. If you average more than 2 iterations per MR, you're on the chopping block as it means you're "sloppy". Its ridiculously dumb. I've heard those numbers matter les…

I've never been seriously threatened with being measured by these metrics, but I have said that if I ever am, my next work task will be some code that takes a single commit and turns it into one commit per line of code changed. If anyone complains, I will take the extra time to turn it into one commit per character of code changed. You want commits? I can give you commits. It's not what I'd really like to do, but by…

I've been thinking about doing something like that. You'd also want to space the commits over the day as well so you can complain about how you were up till 2 working on things and your commit history will bear it out.

Re: The ‘flawed five’ engineering productivity metrics

#50

Post by jqcoffey » Mon, May 16, 2022, 5:43 PM UTC

I’m surprised that folks are still considering metrics like LoC and commit frequency to measure developer productivity, even more so due to the (anecdotal, from my XP of 25 years in industry) fact that as developers gain in seniority they are typically spending more time with people than with code.

IMHO, developer productivity is best judged by the humans they work with.