Live data from Hacker News

Engineering productivity can be measured, just not how you'd expect

okayhq.com

61–70 of 122 posts

Re: Engineering productivity can be measured, just not how you'd expect

#61
post #40

Measuring engineering can be a slippery slope towards stack ranking performance -- which ultimately hurts performance and culture. I think you have to measure with the intent to improve how your team works. If a manager can measure at the team level and open up visibility into the development process, they can hopefully find where things get frustrating (ex. waiting for someone to review a PR). That said, there seem…

I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rv…

> you don't want a single individual to stand out

A study from 2016 at Google discovered that team effectiveness is related to the opportunity for "equal speaking". Similar to your conclusion.

https://www.nytimes.com/2016/02/28/magazine/what-google-lear...

Re: Engineering productivity can be measured, just not how you'd expect

#62
post #40

Measuring engineering can be a slippery slope towards stack ranking performance -- which ultimately hurts performance and culture. I think you have to measure with the intent to improve how your team works. If a manager can measure at the team level and open up visibility into the development process, they can hopefully find where things get frustrating (ex. waiting for someone to review a PR). That said, there seem…

I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rv…

What's missing from your assessment is how good the project itself is.

Maybe Evan is a liability, but he might also be the reason why VueJS is popular and valued in the first place.

It also seems pretty strange to me to count VSCode and GitLab in there, because those are worked on by companies with teams behind them that get paid and which will have constant churn.

If you took VueJS, and had a team at Microsoft take it over from Evan, it too would live on. So I don't know that your explanation for the "risk" here has anything to do with "very high impact" and more to do with a project being maintained by a community of backers, on people's free time, and a project maintained by a company that hires developers to work on it full time.

Re: Engineering productivity can be measured, just not how you'd expect

#63
post #5

Earlier quoted context omitted.

The best way to motivate devs to build truly robust systems is to let them go home at 2pm when things are finished and production process is running smooth. However most organizations would pile more work onto devs if they finish early, so devs then compensate by taking more time on their current tasks. Why finish them early, when you'll be thrown another one right away.

Finish what? Where do you draw the line of “work for today”? I don’t think motivated devs want to go home at 2pm. Speaking as a dev, the best way to motivate me is to give me some head cracking challenge, trust (no reporting) and autonomy (don’t tell me how to do my work).

They might let us go home at 2pm but we can't stop thinking about work problems even in the weekends - that's when we get a nice juicy project to think about.

Re: Engineering productivity can be measured, just not how you'd expect

#64
post #58

One thing that wasn't covered here is that productivity is about: > the effectiveness of productive effort and > the state or quality of producing something Those are quoted from the dictionary definition of productivity, and that definition in my opinion outlines a great insight. Productivity is about the "product" first and foremost. One thing that's often missing is teams don't have ways to quantify how good the p…

Yes! There's a strong tendency toward measuring unvalidated proxies because they are easy or convenient to measure, without even constructing a hypothesis about how these proxies relate to more important things, much less testing it.

Measuring productivity has to start with what matters: do your end-users (these are not necessarily your customers) like what you do? And often it is hard to get this information. Just asking them will lead to biased responses, but there are ways to deal with that.

I happen to be in a business (let's call it food) where I can observe the end user using more of their money with our customers rather than their competitors when we do things right. That's a strong signal -- probably stronger than asking them -- so a very fortunate starting point. (Of course, there are still confounding terms like seasonality, economy etc to grapple with, but there are ways to deal with that too.)

Starting with that one measurement that matters, one can begin exploring proxies. Set up a hypothesis: "Velocity would be easier to measure and the number would be available faster. Does it correlate with our one good metric?" And then you run the experiment. It might take weeks or months to get back the end-user-happiness data that corresponds to this week's velocity, so these tests are expensive (but pay off many times over when you find good proxies.

In the end, you should be able to construct a somewhat sensible model of user happiness, and answer questions like, "if we hire another team member and therefore increase velocity by 3 %, how much happier will our users be? And what is that worth in sales?"

When you can convert everything to the same unit of measurement (dollars are an easily explained option, but log-dollars are a personal favourite of mine) you get intense clarity and alignment around priorities and decisions.

----

All of that said, development speed, as defined in the Accelerate study in particular, is one of those generally good things you pretty much unconditionally want. The reason is given in the study and expanded on further in Reinertsen's Principles of Product Development.

The reason speed is important is that successful product development is controlled by surprises. You will discover something tomorrow that will make you wish you had prioritised differently today, and being able to pivot quickly on surprises is how you both de-fang the biggest risks, but also how you throw yourself at opportunities before your competition even realises there is one.

Re: Engineering productivity can be measured, just not how you'd expect

#65
post #12

The article is correct to point out that in case of no process measurement, politics, not results, will start to dominate. Measuring process inputs will favor Sisyphean work. Appearing to be working, hours punched in and butts-in-seats will be more valued than results. Many companies are stuck in here. Measuring proxies of outputs such as lines of code, or number of tickets closed, as the article mentions, only leads…

I like it too, but I think it misses something crucial: how often is the team unblocking others? This, naturally, is at odds with minimizing interruptions. On one extreme there is no shield around engineers and they experience constant whiplash with "code oracling", shifting priorities, obscure trivialities, and other things that can drive people insane and can prevent meaningful work from being completed. On the oth…

> "code oracling"

Congrats, according to Google it seems like you invented a new word. Could you define it for us?

Re: Engineering productivity can be measured, just not how you'd expect

#66

Engineering activities are economic activities. Engineering activities must make sense as economic activities. Engineering activities productivity can be measured the same way the productivity can be measured for any economic activities. See "Internal Market Economics: Practical Resource-Governance Processes Based on Principles We All Believe In", Book by N. Dean Meyer. The posted article came so close to the idea of…

> Engineering activities productivity can be measured the same way the productivity can be measured for any economic activities.

The economy is basically a population based evolutionary process. Its nature is open ended - things that seem important now might have been seen as being useless or stupid a decade or two ago, like neural nets in 1985 and personal computing in 1950's. So we can't tell ahead of time which will become important, but the exploration process and extreme diversity are the main gain. It's all about getting to those stepping stones that we don't even know we will need. Inspired engineering is eventually rewarded with economic success, just like biological evolution.

Re: Engineering productivity can be measured, just not how you'd expect

#67
post #58

One thing that wasn't covered here is that productivity is about: > the effectiveness of productive effort and > the state or quality of producing something Those are quoted from the dictionary definition of productivity, and that definition in my opinion outlines a great insight. Productivity is about the "product" first and foremost. One thing that's often missing is teams don't have ways to quantify how good the p…

This is nearly always missed in my experience. A good product is "winning" for your engineering team, if you do not have a good product, you have lost. Would you have made a good product if your team spent less time in the locker room(meetings)? Maybe, and certainly more time on the field probably will increase your odds of winning, but it won't ensure it.

This always takes me to USE YOUR PRODUCT. If you don't use your product, you'll never know if it is good. My team builds internal tools for our support folks, we also use these tools every day to try and answer our own questions about internal problems. Are the tools we make getting better? Well, just consult our "How long does it take to answer X?" KPI, we have a set of known common problems; if our support folks are spending less time figuring out the answer to these gold standards, our product is improving, if they're spending longer, we've regressed and we need to change something.

I'll grant that making internal tools grants us a lot of gimmes, we're not tasked with taking advantage of users, and we can spend time training users intensively on new releases, but the underlying principle is the same; a chef will never know if a recipe is good without tasting it himself, and you'll never know if your team is successful if you don't know if the product you've created is good.

Re: Engineering productivity can be measured, just not how you'd expect

#68
post #62
post #40

Earlier quoted context omitted.

I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rv…

What's missing from your assessment is how good the project itself is. Maybe Evan is a liability, but he might also be the reason why VueJS is popular and valued in the first place. It also seems pretty strange to me to count VSCode and GitLab in there, because those are worked on by companies with teams behind them that get paid and which will have constant churn. If you took VueJS, and had a team at Microsoft take…

> So I don't know that your explanation for the "risk" here has anything to do with "very high impact"

When talking about risk, I mean immediate as opposed to long term. Obviously, if you pay people to take over vuejs, they can, provided they are qualified that is. But there is still the ramp up period and the risk of losing undocumented knowledge that Evan has, specifically with his understanding of "what doesn't work".

> It also seems pretty strange to me to count VSCode and GitLab in there, because those are worked on by companies with teams behind them that get paid and which will have constant churn.

The vscode and gitlab project are included because they provide good data points for very fast moving projects, that can be used to help understand closed source software development. Also the development pattern behind vuejs certainly exists in the closed source world as well.

Re: Engineering productivity can be measured, just not how you'd expect

#69

> Just as a sports team wins or loses together, so too should the engineering team be treated as the fundamental unit of success. A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team? Too many times I have see engineering teams as only a team on the org chart In reality they solve tickets as individuals with only a small interac…

> A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team? This is why I like XP. Their teams really are teams like you say. Though I think in many dev shops you can be team like. Someone might like refactoring and cleanup. Someone else is good at rapid prototyping. Another architecture. Sometimes a great dev is just the one who ca…

Ha people ask "who's the best football player" all the time! e.g https://www.pledgesports.org/2017/11/the-10-greatest-nfl-pla...

Re: Engineering productivity can be measured, just not how you'd expect

#70
post #57

> Each engineering team is unique, so its blockers will be specific. It's not so simple as "more maker time is better." If your engineering team is new or temporarily misaligned on key goals, more meeting time might be the answer. What never changes is the need for measurement and well-considered, deliberate decisions And we came full circle back to measuring a teams productivity being an art form. Maybe you have a m…

That’s why they suggest surveys to gauge and looking at how much “maker” time engineers have.
Post reply on HN