Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Worst Software Metrics

gitclear.com

Post reply on HN
Print view View original

61–68 of 68 posts

Re: Worst Software Metrics

#61

Post by overgard »

winrid wrote: post #54
overgard wrote: post #40

Ugh the "velocity" one really hits home. A couple of jobs ago one of our project managers was constantly showing burn down charts (which, of course, never burned down); along with our "velocity". The funny thing is, because we were doing t-shirt-size/fibonacci, our velocity numbers varied so wildly that it was essentially a random number generator. Then we were required fit all our stories within a "budget" of our "a…

To be fair, that's very poorly implemented. You should have reference stories for what is a 1 in complexity and the highest value should be low, like 13, which forces you to break things down and forces the PM to write better specs.

We did all of these things. But we had this ridiculous Fibonacci thing where you would end up with a conversation like this:

dev: well, this looks like it's somewhere between a 5 and an 8, so let's call it 6.

pm: no 6 is not allowed, it must be either a 5 or an 8.

dev: ok, lets call it an 8.

Ok so that's a very plausible conversation, right? I'm sure we've all had it. But just that rule alone made the estimate on this story 20% off, just to satisfy a rule. I'm perfectly fine with that, if we're only talking about that one story. If we're both happy that this might go 20% faster or 20% slower than predicted, no big deal. But if you're going to include it in a larger rollup, you've included a very large error term with your number that you need to carry along with the rest of the calculations. Did you ever take a physics course in college? (A real one -- not one of the baby ones they gave to business and liberal art majors). If they were good, they probably made you deal with carrying an error term in your math. You probably even hated it because it was a pain in the ass to deal with, but it's also kind of important. If you're adding together 50 numbers that are all off by 20%-50% and chucking away the error term, your summary and conclusion are absolute garbage if you do not properly keep the error term included. Math is unforgiving like that. Presenting that absolute garbage as "data" is, in my opinion, professional incompetence. It just is. I guess the softening factor is that so many people do it that it's not an individual incompetence, it's a lack of competence in the profession itself essentially.

Re: Worst Software Metrics

#62

Post by winrid »

overgard wrote: post #61
winrid wrote: post #54

Earlier quoted context omitted.

To be fair, that's very poorly implemented. You should have reference stories for what is a 1 in complexity and the highest value should be low, like 13, which forces you to break things down and forces the PM to write better specs.

We did all of these things. But we had this ridiculous Fibonacci thing where you would end up with a conversation like this: dev: well, this looks like it's somewhere between a 5 and an 8, so let's call it 6. pm: no 6 is not allowed, it must be either a 5 or an 8. dev: ok, lets call it an 8. Ok so that's a very plausible conversation, right? I'm sure we've all had it. But just that rule alone made the estimate on thi…

Well yeah, you don't have to use fib numbers. Not really sure why that's popular anyway.

Personally I prefer being on a small team with excellent devs that don't require this kind of stuff. I built products that made many millions of dollars a year with just me, a PM, and one other dev with none of this stuff.

Re: Worst Software Metrics

#63

Post by namibj »

wbharding wrote: post #8
eirikma wrote: post #4

Does anybody know of a study documenting that measuring and reporting software code quality metrics actually improve the project/product team's ability to deliver? So far I've only found studies documenting that measuring code quality improve the team's ability to improve the measurement figures. That doesn't count. More or less any measurement will be gamed if anyone think they are used for something. But has anybod…

This is a terrific question, and a topic I've thought about quite a bit while we've built our product (context: as the OP, I've toiled on "how to prove metrics are signal-bearing" for a couple years now). The problem we always end up at is "what measurable output can we rely upon to tell the story that a project's fortunes are improving"? To date, its been challenging to find a compelling answer. This may be why I'm…

This sounds like something Felix von Leitner might be interested in. Consider collaborating.

https://blog.fefe.de

Re: Worst Software Metrics

#64

Post by Fire-Dragon-DoL »

overgard wrote: post #42
gtirloni wrote: post #39

Earlier quoted context omitted.

I'm sorry this experience left you scared of metrics but it doesn't mean they are bad, just that people need to know how to use them. I've had this exact conversation before and the extremism of "metrics are always weapons" was a contributing factor for me to leave a, say, less metrics-inclined workplace. Just saying there has to be balance.

I've been a professional software developer for twelve years, and while I'm trying to keep an open mind, I have yet to see any useful metrics for what we do. Like, useful AT ALL. I can think of zero numerical metrics that can accurately tell you what's going on with a team. Number of commits, burndown charts, velocity, issues closed/created, whatever, they're all simultaneously easily game-able and also create bad in…

I'm completely with you on that. As for an example where some metrics might be useful: some form of complexity measurement of the code helps a new developer on the team discover where to poke to find out issues, as well as which parts are critical. While far from perfect, it gives you a starting point that's possibly better than "randomly open files in project".

Now, this comes with interesting points: - the metrics in this case are in the hands of the professional in the field - the metrics are there to serve the person itself, without affecting others

Metrics on commits, line of code, tickets closed are useless from my point of view. If you ask in a developer team who is weak at the moment (assuming there is no room for improvement is stupid) gives better results. A decent programmer is able to spot fake ones and is able to notice performance difference over time (not perfectly, but definitely better than metrics).

Not to mention, I would be surprised if none of the team would start complaining if someone is doing no work and still getting salary.

You can spend an entire day teaching your team something important about the project you are working on, have no commits about it. Or you can use half the day walking around thinking how to solve a bug. Not a single line is written, but that's how some problems are solved.

I started counting in my working hours even when sometimes I wake up in the middle of the night and my mind solves programming problems for 1 hour (let's say). I have a decent employee, being at my desk typing is not what I'm paid for, I'm paid for solving problems with software engineering to the best of my knowledge.

Re: Worst Software Metrics

#65

Post by Fire-Dragon-DoL »

mnowicki wrote: post #52
wbharding wrote: post #12

Earlier quoted context omitted.

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose R…

Please stick with me here, probably going to be a super long post but I'm sharing my thoughts on the whole topic end-to-end with the goal of addressing the root problem here, I agree that such a metric exists which is the same as the desired outcome. Such a metric does, in fact, exist. The metric IS the desired outcome. The problem comes when you try to write a generic, reusable formula that describes the desired out…

Just to highlight this, this is what Toyota does with its engineers (not limited to software).

Managers are engineers of towering technical competence.

Re: Worst Software Metrics

#66

Post by Fire-Dragon-DoL »

overgard wrote: post #59
mnowicki wrote: post #53

Earlier quoted context omitted.

Why is this so true. I usually assume that when a large percentage of people do something that seems dumb, it's usually because there is some reason I don't understand and not because they all coincidentally decided to be idiots in the same way. Occasionally there is a common pitfall that does cause them all to be dumb in the same way, and more often there's a good reason. In this case though, it's so common but I ca…

I think the problem is this: if you talk to any good software developer (or scientist for that matter), they're always going to speak in probabilities and hedge everything. And they're absolutely correct to do this, because it's the reality. The problem is, the business people (generally) have no interest in this. They're interested in certainties. They aren't being irrational either -- they need to go out and sell,…

Stupid question: can't the businessman do its job? I mean, it's not like the market is constant everywhere, they must be dealing with probabilities too.

Re: Worst Software Metrics

#67

Post by ljw1001 »

The 'best' software productivity metric I've ever seen is one I created and tested on a large code base. It only measures coding, but it's better than the other metrics for that. It is based on information theory and credits code removed equally to code added. In the end, I reported the results to the devs and then stopped doing it because there was too much opportunity for abuse by my superiors. Once they got wind of it, they wanted to know everyone's scores and I could see where it was heading.

The metric: It was based on a very powerful (and slow) compression algorithm such that it could measure the change in code. Since it used compression, whitespace, boiler plate, etc had essentially no impact. It was carefully tested by comparing the results against human assessments.

The only clear issue was that it did over-credit mass changes (renaming a root package in a java project which caused all files to be changed). This could probably have been handled by adjusting the scores.

After looking at other metrics (bugs reported, comments on bug reports, code reviews completed, comments in code reviews, etc) it was clear that some people made _huge_ contributions through activities that were ignored by the algorithm, and some people who had great software productivity scores contributed little in these other activities. They were putting all their time into their own code.

FWIW, In the same exercise, I came up with another good metric to identify problematic code (rather than problematic people). That metric used machine learning to find code that needed to be refactored. But that's a story for another time.

Re: Worst Software Metrics

#68

Post by sbellware »

The only telltale metric of software work that I know is the ratio of time spent making progress to time spent not being able to make progress because of past mistakes made and shortcuts taken.

It's reasonable to consider that the reason that software developers are saddled with such shallow process and tooling metrics is that they themselves don't have any better alternatives to offer.

If we roll over and accept the lowest possible maturity organization and process methodologies like Scrum, then we ultimately deserve what we get.

We have a roll to play, and it ultimately requires stepping up and seeing our jobs as more than mere tool-wielding. Or said otherwise, in the rush to call ourselves "engineers" we've stopped being engineers and stopped even pursuing an understanding of the body of knowledge of engineering (which leads to better grasp of the subject area that comprises process and measurement).

Velocity is a standard we're held to because we didn't step up to provide better standards. And we didn't do that because we didn't believe it was our responsibility to learn about the rest of software development. And in that vacuum, people who know even less than us are left to dictate pseudo science to people who should already be in command of the science.

We have a role to play if we wish to play it. It's not obligatory, but there are real costs to pay when we don't.

There's no requirement or expectation to go above and beyond in an effort to catch up on all the knowledge and understanding we've shed from our wheelhouse. It's totally fine and understandable if we don't. But we also have to accept that nature abhors a vacuum.

Post reply on HN