Look at the SPACE framework, its meant as a super() to DORA that is focused on developer productivity vs. systems performance: https://queue.acm.org/detail.cfm?id=3454124
Ask HN: How do you measure eng team productivity?
11–18 of 18 posts
Re: Ask HN: How do you measure eng team productivity?
#12I worked in a company who measured productivity by attributing a score based on the Fibonacci scale. But they were doing this when the feature was done. By doing that, it was fairly easy to give a relevant score to each feature. Engineering team was doing this to measure solely engineering productivity, but product team could do this as well to measure added value to the company. By tracking the start/end date as wel…
That’s very interesting, what is the score based on?
It was supposed to represent "how big" the feature was.
As it's subjective and vague, the score was suggested by the engineer in charge of the feature and then (rapidly) approved by the team during daily meetings.
But I guess this principle could be applied with a score representing whatever you want to track, and however you want to track it.
Re: Ask HN: How do you measure eng team productivity?
#13> However, anything that does not bring obvious value (bug fixing, refactoring) should be counted out of output in terms of productivity Not counting refactoring as output makes sense, because it's more of repositioning for future output, and/or recreation. But if fixing bugs doesn't bring value, why do you do it?
Well, productivity measures how fast you deliver new value. Fixing a bug is about repairing value that you thought you had delivered but actually did not. Counting bugs as "value" would be double counting.
I understand your point, but by doing this you will probably end up with nobody in the team caring about fixing bugs.
That's a problem because sometimes fixing bugs brings more value than adding a new feature or optimizing something.
Re: Ask HN: How do you measure eng team productivity?
#14The organisation is like a complex organism, and engineering is merely piece of the whole. Each piece contributes to and depends on each other (roughly) to make a business function. Therefore, the word "engineering productivity" by necessity should be put under the category of "intangibles" (in most cases).
Still, we can consider some interesting thought experiments. Say, you start with a purely technical team; there are no marketers, no HR, no managers, etc, and the team gets a product going in the market. In such a limited case, one can characterise productivity with "revenue per engineer" (essentially - total revenue / number of engineers).
However, in reality, if the marketers, the HR, and various managers, leaders and salesman don't do their jobs, despite having a great product, revenue will approximate towards zero overtime. In reverse, if the marketers merely do their job perfectly, but there are no engineers delivering the product, again there won't be any profits.
Despite all these caveats, let me share why I think "revenue per employee" seems like the most useful metric here. Consider the "average revenue per employee" for these two companies:
ADP: $284,453
Apple: $2,560,571
Google: $2,020,329
Apple/Google are 10x ADP. I'd argue a good part of this 10x difference we see in the output primarily due to the differences in engineering sophistication. The engineers in these more profitable orgs are doing qualitatively different "types" of engineering (hardware? prototyping? deep research? higher standards?). Increasing engineering prowess, trying more sophisticated projects seems like the biggest boost to the bottom line.
Re: Ask HN: How do you measure eng team productivity?
#15> However, anything that does not bring obvious value (bug fixing, refactoring) should be counted out of output in terms of productivity Not counting refactoring as output makes sense, because it's more of repositioning for future output, and/or recreation. But if fixing bugs doesn't bring value, why do you do it?
Well, productivity measures how fast you deliver new value. Fixing a bug is about repairing value that you thought you had delivered but actually did not. Counting bugs as "value" would be double counting.
> Fixing a bug is about repairing value that you thought you had delivered but actually did not.
In that case it's important to classify issues correctly. Email sending worked yesterday but not today because Office365 changed some SMTP requirements (say ciphers)?
Customer will call it a bug, but according to your definition it should be classified as a feature.
Re: Ask HN: How do you measure eng team productivity?
#16Why do you want to measure productivity? You already know where you are. A measurement doesn’t help your team ship better or faster. If you’re shipping product improvements at a regular cadence, and the customers are happy, then there’s no problem with productivity. If you’re shipping poor quality, identify why and improve that. If you’re shipping so slow that it’s negatively impacting customers, identify why and imp…
Interesting to note that two of your decision points are on "shipping product improvements at a regular cadence" or "shipping so slow that it's negatively impacting customers". Both relate to a productivity discussion, ie "how to define what it means to 'ship' as an engineering team". In that sense, we're aligned on why it's important to measure productivity. It would however be a mistake to try to optimize for maxim…
The material output of programmers is code. In that sense, programmer productivity is (|additions| + |deletions|) / time. Economically, it’s (|additions| + |deletions|) / wages paid.
This is how I would quantitatively measure the productivity of members on the teams I work with. If you look at this measure across your team, I think you’ll find it roughly matches your intuitive notion of who the most productive programmers are. I also think you’ll find story size positively correlated with the magnitude of code change or time.
The thing is, measuring how fast programmers output software isn’t all that useful.
Re: Ask HN: How do you measure eng team productivity?
#17Earlier quoted context omitted.
Interesting to note that two of your decision points are on "shipping product improvements at a regular cadence" or "shipping so slow that it's negatively impacting customers". Both relate to a productivity discussion, ie "how to define what it means to 'ship' as an engineering team". In that sense, we're aligned on why it's important to measure productivity. It would however be a mistake to try to optimize for maxim…
The material output of programmers is code. In that sense, programmer productivity is (|additions| + |deletions|) / time. Economically, it’s (|additions| + |deletions|) / wages paid. This is how I would quantitatively measure the productivity of members on the teams I work with. If you look at this measure across your team, I think you’ll find it roughly matches your intuitive notion of who the most productive progra…
This feels like you're measuring lines of code as productivity, but most experienced engineers would disagree that lines of code is a good metric. Am I misunderstanding your assertion?
Re: Ask HN: How do you measure eng team productivity?
#18Earlier quoted context omitted.
The material output of programmers is code. In that sense, programmer productivity is (|additions| + |deletions|) / time. Economically, it’s (|additions| + |deletions|) / wages paid. This is how I would quantitatively measure the productivity of members on the teams I work with. If you look at this measure across your team, I think you’ll find it roughly matches your intuitive notion of who the most productive progra…
This feels like you're measuring lines of code as productivity, but most experienced engineers would disagree that lines of code is a good metric. Am I misunderstanding your assertion?
Productivity is the ratio of output to input.
Code is the output of a programmer. Time (or wages) is the primary economic input to software development.
Productivity is not a measure of whether something is worth doing. It says nothing about whether the code is valuable or worth writing. A factory has increased productivity if it has produced more widgets with the same inputs, even if nobody wants those widgets or those widgets pollute the environment.
Similarly, programmers do not choose what software to write. They are employed to write software someone else decides is valuable to the business. Their productivity is independent of whether the software has business value.
Consider a one line program. A programmer that produces that program in 5 minutes is more productive than one who produces it in 1 hour.
Stories with sizes are proxies for magnitude of code change or time. A larger project is larger because it takes more time or more code change, or both. Story size/points is an attempt to estimate this. So it is more accurate to measure the actual code change and time taken.
It’s true that the same program can be written with different amounts of code. But this is largely irrelevant at the organization level, where programmers are not writing multiple versions of same program in different languages and toolsets. A team is typically writing software within the context of a particular toolchain and runtime environment.