Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineering productivity can be measured, just not how you'd expect

okayhq.com

Post reply on HN
Print view View original

101–110 of 122 posts

Re: Engineering productivity can be measured, just not how you'd expect

#101

Post by michaelt »

sdesol wrote: post #40
necco908 wrote: post #34

Measuring engineering can be a slippery slope towards stack ranking performance -- which ultimately hurts performance and culture. I think you have to measure with the intent to improve how your team works. If a manager can measure at the team level and open up visibility into the development process, they can hopefully find where things get frustrating (ex. waiting for someone to review a PR). That said, there seem…

I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rv…

> Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk.

If you're trying to sell this concept to developers, you might want to change "discourage" to "hire more than one"

Re: Engineering productivity can be measured, just not how you'd expect

#102

Post by marcus_holmes »

andrey_utkin wrote: post #88
marcus_holmes wrote: post #82

Earlier quoted context omitted.

Engineer A built the new widget that Sales have been talking about for months. This immediately generates $1m in revenue. Engineer B removed some tech debt. Future efforts to build widgets will take less time to build. Engineer C rewrote the backend to prevent a vulnerability exposure that could have lead to disaster. Who was most productive? Who deserves a raise?

Each of these tasks, or ongoing commitments, has a clear price when you run an organization the Internal Economics way. Whoever consistently "earns" more than their salary this way, deserves a raise.

Ok, but some tasks are less valuable and take more time to do, but are completely necessary. How do you value something like that? What if the "value" is actually less than the cost of the task (but it's still absolutely necessary)?

And if a task has an internal price on it, but the development time slips, to the point it exceeds the price, do you stop development?

I think this method is interesting, but I have a feeling that all you're doing is shifting the complexity around. The underlying complexity is still there (it's impossible to accurately estimate a development task and impossible to measure developer productivity). I guess that's a good thing, though - now the whole company has to deal with the complexity rather than just the dev team ;)

Re: Engineering productivity can be measured, just not how you'd expect

#103

Post by andrey_utkin »

marcus_holmes wrote: post #102
andrey_utkin wrote: post #88

Earlier quoted context omitted.

Each of these tasks, or ongoing commitments, has a clear price when you run an organization the Internal Economics way. Whoever consistently "earns" more than their salary this way, deserves a raise.

Ok, but some tasks are less valuable and take more time to do, but are completely necessary. How do you value something like that? What if the "value" is actually less than the cost of the task (but it's still absolutely necessary)? And if a task has an internal price on it, but the development time slips, to the point it exceeds the price, do you stop development? I think this method is interesting, but I have a fee…

> What if the "value" is actually less than the cost of the task (but it's still absolutely necessary)?

This is a false premise. But it's surprising how many people seem to hold it, to their peril.

> I have a feeling that all you're doing is shifting the complexity around. The underlying complexity is still there

That's right, but do you agree this approach moves it to a place where it makes more sense, where it informs good decisions and is manageable?

> it's impossible to accurately estimate a development task and impossible to measure developer productivity

It's not impossible, but it's not something we as a society or an industry have a common fine grasp on. On this topic, I like best the books by Doug Hubbard: "How to measure anything" and "The failure of risk management: what is it and how to fix it".

It just requires yet another unusual mindset: probabilistic thinking, in addition to the above-established value-based thinking. You have to use a technique called calibrated probability assessment. We started practicing this at my workplace, and it seems to be working as intended, but we're not well calibrated yet.

Re: Engineering productivity can be measured, just not how you'd expect

#104

Post by blackbear_ »

andrey_utkin wrote: post #98
blackbear_ wrote: post #93

Earlier quoted context omitted.

The key question is how do you quantify earnings/savings? In the example: Future efforts to build widgets will take less time to build -> How to quantify ? Vulnerability exposure that could have lead to disaster -> How to quantify ? What you are saying is nice and clean on paper but simply impossible in practice.

It's an investment problem for those who hold the money in the organization. Say, we have an organisation consisting of exec team and engineering team. Engineering team has a bright idea that a certain undertaking will reduce time to make one widget. They bring this case to exec team. "It will reduce the cost of a widget from X1 to X2, the project will take T time and cost C." If exec team sees this investment intere…

Yes. Again, properly estimating X1, X2, T and C is so difficult in most cases that this strategy is not really applicable (notable example: technical debt).

Re: Engineering productivity can be measured, just not how you'd expect

#105

Post by andrey_utkin »

blackbear_ wrote: post #104
andrey_utkin wrote: post #98

Earlier quoted context omitted.

It's an investment problem for those who hold the money in the organization. Say, we have an organisation consisting of exec team and engineering team. Engineering team has a bright idea that a certain undertaking will reduce time to make one widget. They bring this case to exec team. "It will reduce the cost of a widget from X1 to X2, the project will take T time and cost C." If exec team sees this investment intere…

Yes. Again, properly estimating X1, X2, T and C is so difficult in most cases that this strategy is not really applicable (notable example: technical debt).

See http://www.hubbardresearch.com/wp-content/uploads/2019/06/In...

Re: Engineering productivity can be measured, just not how you'd expect

#106

Post by thinkersilver »

horsawlarway wrote: post #17
bjornsing wrote: post #8

> Engineers will become more focused and engaged, managers will become more effective and empathetic, and companies will build faster with higher quality. Engineering will rise to a whole new level. A bit too much hyperbole for my taste, given the less than groundbreaking ideas.

I mostly agree - This doesn't seem to be adding any real visibility that velocity tracking (a la - agile) wouldn't give you already (not that I'm advocating for agile, mind you...). Consider - I have two teams, with the same staffing levels and the same general seniority. For this example, lets assume each is a team of 5, with 1 tech lead, 2 seniors, and 2 juniors. Both teams have the same approximate meeting count,…

It helps to instrument the journey of your work from jira all the way to build, deployment, run and monitoring (observation).

From that you can get measurements on how long each stage takes and the duration of each transition.

From there you can compare team A and B. The transition times is where the human time cost usually sits.

Just getting the time when a Jira or feature is raised, to the time it is picked up to the time of the first commit to the time of the first test and final build does already give you valuable insight.

The points you raised towards the end can be answered if observability of your CI/CD pipeline is actually in place or at least a place to start a line of inquiry.

Naturally the blockers will be aggregated into some of the values but as you work through the journey, they will start clustering at certain stages and maybe highlight a significant problem that needs to be addressed.

There's a wealth of data being left on the table that can help inform management decisions.

Re: Engineering productivity can be measured, just not how you'd expect

#107

Post by 908B64B197 »

dacracot wrote: post #39

> Just as a sports team wins or loses together, so too should the engineering team be treated as the fundamental unit of success. A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team? Too many times I have see engineering teams as only a team on the org chart In reality they solve tickets as individuals with only a small interac…

>> Just as a sports team wins or loses together, so too should the engineering team be treated as the fundamental unit of success.

> A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team?

It's a great analogy, but the author should also keep a couple things in mind:

- Out of all the football teams in the world, only 32 can ever win the SuperBowl

- Most of the money is in the SuperBowl (winner takes all)

- Professional athletes are extremely well coached and compensation is extremely competitive

- Professional athletes only train and play: there's support staff for everything else

Re: Engineering productivity can be measured, just not how you'd expect

#108

Post by NotPavlovsDog »

michaelt wrote: post #100
NotPavlovsDog wrote: post #96

Earlier quoted context omitted.

That's what you discuss in the organization.

What were the conclusions of that discussion in your organisation?

See my parent comment. One of the agreements we arrived at was not to have non-product-facing teams.

The teams also decided on how and who would be allocated to the tasks that could be placed as similar to what you describe.

Re: Engineering productivity can be measured, just not how you'd expect

#109

Post by blackbear_ »

andrey_utkin wrote: post #105
blackbear_ wrote: post #104

Earlier quoted context omitted.

Yes. Again, properly estimating X1, X2, T and C is so difficult in most cases that this strategy is not really applicable (notable example: technical debt).

See http://www.hubbardresearch.com/wp-content/uploads/2019/06/In...

Calibrated estimates with huge uncertainty ranges are still useless, even if calibrated.

"One month of refactoring this component can save us between 500k and 50M in the next five years with 90% probability" - this is not very useful for the decision maker, yet it is quite difficult for a domain expert to narrow the range.

Re: Engineering productivity can be measured, just not how you'd expect

#110

Post by GizmoSwan »

GizmoSwan wrote: post #79

Good work. That is how it should be measured. In economics as far as I know, they measure productivity via salaries. The more money people make, the more productive they are assumed to be!

'Evaluations of an economy's productivity performance are made using a measure of real gross domestic product (GDP), which represents the constant dollar income (labour income plus profits) that an economy generates through domestic production, with the volume or constant dollar indices being calculated from the prices'

'DP=(GDP/Hours)*(Hours), (1)

where Hours is the total number of worker-hours.

Chart 1 depicts changes in each of these components over time. For the entire 1961-to-2012 period, labour productivity advanced at a 1.9% annual average, accounting for slightly more than half of the increase in GDP growth. The rest is attributed to hours, which increased at 1.5% per year on average.

Aggregate GDP measures the returns to both labour and capital. Distributional concerns lead to questions about whether the share going to labour increases over time and, in particular, how productivity growth is related to real income.'

https://www150.statcan.gc.ca/n1/pub/15-206-x/15-206-x2014038...

Post reply on HN