Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Worst Software Metrics

gitclear.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Worst Software Metrics

#21

Post by plorkyeran »

wbharding wrote: post #12
Kinrany wrote: post #7

Earlier quoted context omitted.

And at the same time you still can't use LoC to measure productivity and reward for it because of the Goodhart's law.

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose R…

If you reward a team for increasing story points completed, then you will very quickly find that every single story has the maximum possible estimate. In the longer term you'd get stories split into more smaller stories far more often than is reasonable.

It's an incredibly obvious example of a measure that's useful only so long as there's no incentive to game it.

Re: Worst Software Metrics

#23

Post by TuringNYC »

sebastianconcpt wrote: post #18

Yep! Considering that the metrics above are still considered viable at the dawn of 2020 proves how much opportunity still remains for managers to improve the measurement -> incentives -> long-term outcomes achieved by their team. To their credit, the instinct that leads managers to utilize these metrics is sound. They know that the best businesses make their best decisions using data. But so much depends upon whether…

>> The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.

Only if you are not being rewarded for solving your own bugs. Otherwise, I've seen this go horribly awry.

I was at a Big 5 consulting company where one project was big on "how well you managed issues and brought them to resolution". That meant that good developers who prioritized code coverage and mostly bug-free code did not have good issue resolution case studies to cite during promotion cycles. After a year of good developers getting dinged on this, some good developers banded together to purposefully leave known bugs in the release (especially bugs that could be blamed on vague specs -- so instead of trying and resolve issues during dev cycles, they were propagated to production). Then, the developer would swoop in and do lots of "investigation", produce a fancy write-up and email (carefully cc'ing senior management) and throw the business analyst under the bus.

Senior Management naturally was impressed and some of the developers got promoted for their "leadership" and management of "critical bugs."

Bad Incentives --> Bad Outcomes

Re: Worst Software Metrics

#24

Post by jghn »

wbharding wrote: post #12
Kinrany wrote: post #7

Earlier quoted context omitted.

And at the same time you still can't use LoC to measure productivity and reward for it because of the Goodhart's law.

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose R…

It wasn’t that long ago that OPS was not rewarded, and batting average was king. It took a lot of energy to change that tide and reward players for focusing on other metrics.

My point here is that Goodhart’s law still held, but the fallacy is when people assume that metrics are static.

Re: Worst Software Metrics

#25

Post by ratww »

plorkyeran wrote: post #21
wbharding wrote: post #12

Earlier quoted context omitted.

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose R…

If you reward a team for increasing story points completed, then you will very quickly find that every single story has the maximum possible estimate. In the longer term you'd get stories split into more smaller stories far more often than is reasonable. It's an incredibly obvious example of a measure that's useful only so long as there's no incentive to game it.

I saw that happen in a team whose OKR was stupidly tied to story points. They went from 20 points per sprint to 70 overnight, by splitting stories as most as possible. It was hilarious when they reached their quarter OKR in less than a month.

Re: Worst Software Metrics

#26

Post by dkersten »

toolslive wrote: post #2

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

Well, when comparing languages, sure. 10 lines of python gets a lot more done than 10 lines of assembly, but as a metric of how good your code is or the developer is, it’s not so great. Usually 9 lines of python is better than 10 lines of python (assuming lines means code points or statements or whatever rather than physical lines, otherwise 10 lines could be better if it’s visually laid out clearer)

Re: Worst Software Metrics

#27

Post by forgotmylogin2 »

Is this operating under the assumption that there is no code review?

With code review, lines of code and commit count become much harder to game. If you try to submit changes with unnecessary whitespace, unnecessary code constructs, or incomplete code, your reviewer will see it and tell you to fix it. Ultimately, trying to increase your lines of code or commit count will come back to haunt you as you spend much more time addressing comments from code review.

Re: Worst Software Metrics

#28

Post by abathur »

ilaksh wrote: post #15

I have a solution to these metric gotchas. Don't install Jira at all and then you won't be tempted by any of them. Better yet, don't hire a manager at all. Instead get one or two senior devs and include them in some business discussions and incentives.

I wasn't going to say this quite the same way, but I think I was going to say roughly the same thing.

The metrics are all an effort to abstract developer output into something a manager with no/minimal domain experience can understand.

It feels very similar to the measurement/quantification problems distorting academic output.

Re: Worst Software Metrics

#29

Post by ben509 »

wbharding wrote: post #12
Kinrany wrote: post #7

Earlier quoted context omitted.

And at the same time you still can't use LoC to measure productivity and reward for it because of the Goodhart's law.

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose R…

I think your counterexamples reveal why those metrics work:

* In baseball, batting average and OPS require you to compete against other teams.

* In finance, ROI is tested in the marketplace.

* In sales, conversions require you to convince a skeptical customer.

The software metrics are all things we have fairly direct control over.

Maybe that suggests an amendment to Godhart's law:

> When a measure that is controlled by the measuree becomes a target, it ceases to be a good measure.

Re: Worst Software Metrics

#30

Post by shakezula »

ilaksh wrote: post #15

I have a solution to these metric gotchas. Don't install Jira at all and then you won't be tempted by any of them. Better yet, don't hire a manager at all. Instead get one or two senior devs and include them in some business discussions and incentives.

This is, in my opinion, the actual best approach.

Whenever I see a manager reaching for a "code metric", my immediate thought is that they're so far removed from the actual engineering of the product that they don't feel like they have any feedback so they want to reach for something to feel like they do.

Post reply on HN