Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Worst Software Metrics

gitclear.com

Post reply on HN
Print view View original

11–20 of 68 posts

Re: Worst Software Metrics

#11

Post by jasonpeacock »

eirikma wrote: post #4

Does anybody know of a study documenting that measuring and reporting software code quality metrics actually improve the project/product team's ability to deliver? So far I've only found studies documenting that measuring code quality improve the team's ability to improve the measurement figures. That doesn't count. More or less any measurement will be gamed if anyone think they are used for something. But has anybod…

Check out the book "Accelerate", it answers your questions about which metrics matter, and how.

Re: Worst Software Metrics

#12

Post by wbharding »

Kinrany wrote: post #7
toolslive wrote: post #2

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

And at the same time you still can't use LoC to measure productivity and reward for it because of the Goodhart's law.

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples:

* In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it.

* In finance, ROI is rewarded, and investors whose ROI is higher will generally be more successful

* In sales, "conversions" is a useful metric that salespeople strive toward and are validly rewarded for improving

* In software, if a metric like "story points" (as decided by group via planning poker) could be generated consistently for every issue worked on, there's no apparent reason that developers or teams would be worse off if developers used "story points completed" as a goal to strive toward

I suspect the prevalence of folks citing Goodhart's law is because it is usually true. But this may just be a consequence of the frequent disparity between "metrics available" and "desired underlying phenomenon." There's no fundamental reason that they can't be the same.

Re: Worst Software Metrics

#13

Post by kabdib »

toolslive wrote: post #2

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

I've seen some of the worst developers commit more LOCs than others (churn in badly written code, or "cut-and-paste hell"). I've also seen developers sandbagging the metric (as well as artificially inflating code coverage measurements) by checking in fluff code; a null check on every pointer or reference parameter can get you 5-10 lines of actual live and executable code that is practically guaranteed bug-free and politically correct, even for functions you haven't really started implementing yet. Saw this being used in a USB stack once, it had been worked on by serial contractors and was a real eye-opener (when we got it, it was 5K LOC or so; we whacked it down to about 1100 LOC by removing a bunch of badly-motivated layers; removing the junk sped it up quit a bit, too).

My position is that if you're evaluating people by comparing metrics, you're probably a terrible manager.

Re: Worst Software Metrics

#14

Post by lalaithion »

x3n0ph3n3 wrote: post #10
toolslive wrote: post #2

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

> people can produce & maintain software at about 10 LOC/h. That is shocking and horrifying if that's the norm for software development.

Over the last year, I have averaged 8.95 LOC* per hour as a full time* developer.

Lines of Java + TypeScript, my two most used languages, purposefully excluding configs, html/css, etc. Assuming 8 hours of work per day; this is less than the time I spend at work, but probably more than I spend coding, due to meetings/email/overhead. 246 workdays, which is 30 days off, some of which are public holidays and the rest are vacation. I'm a relatively junior dev.

Re: Worst Software Metrics

#15

Post by ilaksh »

I have a solution to these metric gotchas. Don't install Jira at all and then you won't be tempted by any of them. Better yet, don't hire a manager at all. Instead get one or two senior devs and include them in some business discussions and incentives.

Re: Worst Software Metrics

#16

Post by raxxorrax »

toolslive wrote: post #2

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

But if you measure it to determine performance, you are not paying developers to create solutions, you are paying them to create LOC. That could have quite a negative influence.

If we could channel energy from creating performance metrics into a form of reporting that continually narrows down the problems that need solving, productivity would be served better.

Otherwise we could probably use keystrokes per second just as well.

Re: Worst Software Metrics

#17

Post by greenshackle2 »

x3n0ph3n3 wrote: post #10
toolslive wrote: post #2

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

> people can produce & maintain software at about 10 LOC/h. That is shocking and horrifying if that's the norm for software development.

I agree, that's probably too many lines, considering how crappy most software is.

Re: Worst Software Metrics

#18

Post by sebastianconcpt »

Yep!

Considering that the metrics above are still considered viable at the dawn of 2020 proves how much opportunity still remains for managers to improve the measurement -> incentives -> long-term outcomes achieved by their team. To their credit, the instinct that leads managers to utilize these metrics is sound. They know that the best businesses make their best decisions using data. But so much depends upon whether the data is any good. When you place trust in a metric like commit count, you're liable to end off worse than if you'd used no data at all (i.e., by pitting the team against one another).

Our opinion is that a less bad metric would be one that harnessed the signal lurking deep within lines of code, while rinsing away its noise. The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.

Re: Worst Software Metrics

#19

Post by alexott »

toolslive wrote: post #2

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

It’s really a complicated topic - is it a new project vs. maintenance, etc. In new project my metric was around 500 LOC of C++ code (production ready) per day. In maintenance, it could be a line a day (if you’re lucky).

Re: Worst Software Metrics

#20

Post by iudqnolq »

wbharding wrote: post #12
Kinrany wrote: post #7

Earlier quoted context omitted.

And at the same time you still can't use LoC to measure productivity and reward for it because of the Goodhart's law.

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose R…

I think a better version is "you get what you measure".

Metrics fail when what they measure isn't directly related to what you want. I'd much rather have n conversions than n000 LoC, if all I get to pick is n.

(I suspect this is especially an issue when you find a metric that retroactively correlates reasonably well with something desired. For example, you find that closing a ticket in less than 20 minutes correlates with the issue getting an incomplete solution. Then you make it into a metric, at which point you're bitten by the fact the correlation no longer works: people just leave the ticket open in a tab for a few hours, ignoring it but letting the clock run, before they close it. It's not enough for a metric to historically correlate with good things, "gaming" the metric has to cause the good things as well. Obligatory relevant XKCD (the helpful comments one): https://xkcd.com/810/)

Post reply on HN