Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is developer productivity and how to measure it?

axolo.co

Post reply on HN
Print view View original

71–80 of 91 posts

Re: What is developer productivity and how to measure it?

#71

Post by thenerdhead »

passivate wrote: post #65
thenerdhead wrote: post #35

> Each organization can set a wide range of metrics to follow every week, such as: > Number of commits. > Average commit size. > Frequency of code reviews. > Number of code reviews. > Time to review. > and so on... No. This has been tried many times and companies think this is how you measure productivity, but it is not even sustainable. Developer productivity is not about moving the needle, it is about outcomes, and…

>Developer productivity is not about moving the needle, it is about outcomes, and not outputs. What criteria does your team use to measure the outcomes and/or during a post-mortem? > The real goal is to be more like a f1 pit crew where you leave out the metrics and end up performing better than if you measured them. But all F1 pit crews have defined measurable metrics for success. I don't see the analogy here? Can yo…

We measure happiness and various sentiments towards these areas like knowledge transfer, documentation, ci/cd reliability, pr velocity, etc.

The f1 pit crew analogy refers to how certain teams realized that measuring KPIs too religiously meant the vary difference of performance when it mattered.

Similar to other sports where coaching actually matters and these metrics are rather useless unless there’s a coaching role to be deliberate with them.

Re: What is developer productivity and how to measure it?

#72

Post by tharkun__ »

giantg2 wrote: post #51
marcinzm wrote: post #45

Earlier quoted context omitted.

Everything including "objective" metrics includes bias. And that's before you take into account people outright gaming metrics (objective or subjective).

As a simplified example, if I write 1000 lines of code and you write 1000 lines of code. We should have the same rating if that's the metric used. There shouldn't be any bias there. It only introduces bias when the manager feel your code is better than mine, etc. Now the objective measure itself might have some sort of bias, but at least the rules are set and you're not getting screwed over based on someone's feeling…

In order for these metrics to have even a tiny tiny chance of not being completely gamed (even unintentionally) you'd have to define a rigorous formula of weighted metrics that take things like one of my siblings mentioned into account (did your 1000 lines create a regression or 5 and mine didn't? Code quality? Lots of review comments that took forever to debate and resolve?). And that's assuming you could actually measure those things properly. Was that comment a valid one regarding you missing quality guidelines or was it someone trying to game your metrics negatively so that he'd look better?

I think it's impossible to create something like that and it'd be very very bureaucratic and still prone to gaming. I think having something 'in between' is the best approach. You still allow a manager to interpret these things together with you but the manager should give you a guideline for what to look out for. We can use these metrics to inform decisions about performance but it's completely counter productive to simply have a few metrics where you have to hit specific numbers.

PR throughput? No problem, I'll form a clique of a bunch of people that OK each others tiny PRs. This will result in so much overhead that we won't actually get much done, piss off other team members, create a hell of basically unusable commits, make it more likely that code quality suffers because nobody has any chance of having an overview of what you're doing overall and you will likely create regressions that developed over multiple commits and would've been caught otherwise because let's face it, each unit test you write is its own PR. You say obviously you won't get through with this because your manager is supposed to stop that? Well he can't if we just want a completely objective and metrics driven approach!

With the hybrid approach, you know from your manager that PR throughput is important not at the expense of quality and other things. You want small PRs for certain reasons but not at all costs. There is no exact formula because no two situations are exactly the same. Of course bias comes, of course bad managers make this bad. So does a completely "objective" metrics driven environment in which you play the metrics game. There is no perfect solution.

Re: What is developer productivity and how to measure it?

#73

Post by PeterisP »

giantg2 wrote: post #51
marcinzm wrote: post #45

Earlier quoted context omitted.

Everything including "objective" metrics includes bias. And that's before you take into account people outright gaming metrics (objective or subjective).

As a simplified example, if I write 1000 lines of code and you write 1000 lines of code. We should have the same rating if that's the metric used. There shouldn't be any bias there. It only introduces bias when the manager feel your code is better than mine, etc. Now the objective measure itself might have some sort of bias, but at least the rules are set and you're not getting screwed over based on someone's feeling…

The thing is that those metrics are very poor metrics that don't correlate well to the "true ideal performance", even if compared to a subjective manager's intuition with all the randomness and biases.

Replacing a subjective metric that's at least somewhat effective with a metric that's totally useless (because those inherent inaccuracies/biases are even worse than even a poor manager's judgement), that's throwing out the baby with the bathwater. The primary purpose of a performance metric is to measure performance, and being prejudice-resistant is something that's nice to have - the primary reason why you implement a metric is not because you need something that can be argued.

Re: What is developer productivity and how to measure it?

#74

Post by passivate »

thenerdhead wrote: post #71
passivate wrote: post #65

Earlier quoted context omitted.

>Developer productivity is not about moving the needle, it is about outcomes, and not outputs. What criteria does your team use to measure the outcomes and/or during a post-mortem? > The real goal is to be more like a f1 pit crew where you leave out the metrics and end up performing better than if you measured them. But all F1 pit crews have defined measurable metrics for success. I don't see the analogy here? Can yo…

We measure happiness and various sentiments towards these areas like knowledge transfer, documentation, ci/cd reliability, pr velocity, etc. The f1 pit crew analogy refers to how certain teams realized that measuring KPIs too religiously meant the vary difference of performance when it mattered. Similar to other sports where coaching actually matters and these metrics are rather useless unless there’s a coaching role…

>We measure happiness and various sentiments towards these areas like knowledge transfer, documentation, ci/cd reliability, pr velocity, etc.

I see, so you guys don't measure anything specific that is actionable? Did you face any challenges dealing with non-performers or under-performers dragging the team down?

Re: What is developer productivity and how to measure it?

#75

Post by closeparen »

giantg2 wrote: post #51
marcinzm wrote: post #45

Earlier quoted context omitted.

Everything including "objective" metrics includes bias. And that's before you take into account people outright gaming metrics (objective or subjective).

As a simplified example, if I write 1000 lines of code and you write 1000 lines of code. We should have the same rating if that's the metric used. There shouldn't be any bias there. It only introduces bias when the manager feel your code is better than mine, etc. Now the objective measure itself might have some sort of bias, but at least the rules are set and you're not getting screwed over based on someone's feeling…

But that is proper. The quality of craft/creative work matters. It belongs in the evaluation of craftsmen and creative workers. And it is fundamentally a feeling. When you are junior you may not yet have developed this judgement or taste. Your job is to learn it, from your own failures and the feedback of your senior colleagues. When you are senior, you have it. You are more valuable to an organization precisely because you can be trusted to have positive feelings about good work and negative feelings about bad work, and therefore do the right thing in a position of decision-making power. Also because you enculturate the next generation of senior craftsmen through your feedback.

This shouldn't be surprise at performance review time, nor should it necessarily come from your manager -- it should be coming from your senior colleagues on each of your code reviews, giving you a chance to improve your bad code before it gets checked in. But when your senior colleagues think your PRs are worse on average than those of your peers, then yes absolutely you should get a worse rating.

Re: What is developer productivity and how to measure it?

#76

Post by closeparen »

passivate wrote: post #66
jgust wrote: post #33

Earlier quoted context omitted.

There's an entire class of products I'll name "internal platform tools" whose primary objective is to improve the developer experience with the intent of having the side effect of increased developer productivity by making it easier & more enjoyable to build things within a company. The teams working on these tools need to understand how their products perform the same as a team building some widget for a "paying" cu…

Agreed, we need both quantifiable metrics, and also a human brain to interpret them with subjectivity, context and compassion. I see many people wanting to take writing code into the liberal arts domain, but I am of the opinion that it may be more useful if we can overlap it with the engineering domain. IMHO the goal should be to repeatedly churn out high-quality bug-free code, and to create an objective process meth…

The part where the code needs to execute correctly is engineering domain. But lots of bad code executes correctly. "Programs must to be written for people to read, only incidentally for machines to execute." Writing things for people to read is unavoidably an arts discipline.

Re: What is developer productivity and how to measure it?

#77

Post by closeparen »

qbasic_forever wrote: post #70
marcinzm wrote: post #62

Earlier quoted context omitted.

Your comment doesn't change if you replace lines of code with manager's perception of you. If you're both equally liked by your manager then you should receive the same rating. Within the metric being defined neither is biased since they have clear and explicit definitions. Against the true metric of "productive engineer" both are biased.

And how do you handle your manager having a cultural or unconscious bias against your ? You just live with them not liking you and not measuring up to others in their mind?

Subjective evaluation for performance purposes is often done by committee for this reason. Your work is read by several people who are unlikely to have the same idiosyncratic biases, at least some of whom don't know you. (That cuts both ways, though; they also don't know the context for the work).

Re: What is developer productivity and how to measure it?

#78

Post by lacker »

giantg2 wrote: post #68
bluefirebrand wrote: post #56

Earlier quoted context omitted.

I don't think adding metrics is actually a good approach to reduce bias. Any kind of measurement can be twisted if you want to. Also, the metrics you choose will undoubtedly contain a measure of your bias anyways. For instance, the metrics two people would choose to represent developer effectiveness will not be the same, and those differences will reveal what kinds of workers they prefer.

I prefer some kind of metric because I'm tired of screwed over by blind shitty managers.

I hear you. But metrics aren't going to save you. You need to find a manager who isn't shitty! They are out there, don't lose hope.

Re: What is developer productivity and how to measure it?

#79

Post by tonyedgecombe »

lacker wrote: post #78
giantg2 wrote: post #68

Earlier quoted context omitted.

I prefer some kind of metric because I'm tired of screwed over by blind shitty managers.

I hear you. But metrics aren't going to save you. You need to find a manager who isn't shitty! They are out there, don't lose hope.

There aren't many though and almost none who haven't come up through the trenches themselves.

Re: What is developer productivity and how to measure it?

#80

Post by tonyedgecombe »

kelnos wrote: post #58
lacker wrote: post #3

I don't agree with most of the advice in this article but rather than complain let me suggest an alternative. As a line manager, with software engineers reporting directly to you, you should be able to use your personal judgment to understand the productivity of your software engineers. Don't measure it with acronyms, with metrics like the number of commits, or by paying attention to how many hours a week people are…

Absolutely agree. I think there is a conflict of interest there, though. Managers have a vested interest in saying that their team is highly productive. Managers of highly productive teams get raises and more head count, and eventually promotions. Anything else reflects poorly on the manager. So the managers-of-managers do need to keep their eyes on this too, but I also agree with you that it's harder for people in t…

Measuring team output seems just as difficult as for an individual.

I was reading an interesting piece in the economist today about maintaining peoples performance on a long space flight. One part that stood out to me is people being productive makes them happy. I always assumed the causation would be the opposite way round. Perhaps the best a manager of managers can do is try and figure out if the members of each team are happy or not.

Post reply on HN