Show HN: Weave - actually measure engineering productivity • Hacker News Board

Re: Show HN: Weave - actually measure engineering productivity

#21

Post by adchurch » Wed, Nov 20, 2024, 9:56 PM UTC

jumploops wrote: post #14

adchurch wrote: post #3

Our metric is approximately "hours of work for an expert engineer." Here are some example open source PRs and their output metrics calculated by our algorithm: https://github.com/PostHog/posthog/pull/25056 : 15.266 (Adds backend, frontend, and tests for a new feature) https://github.com/microsoft/vscode/pull/222315 : 8.401 (Refactors code to use a new service and adds new tests) https://github.com/facebook/react/pull…

Curious how these numbers correlate to the estimates of the engineers behind the PRs? For example, the first PR is correlated with ~15 "hours of work for an expert engineer" Looking at the PR, it was opened on Sept 18th and merged on Oct 2nd. That's two weeks, or 10 working days, later. Between the initial code, the follow up PR feedback, and merging with upstream (8 times), I would wager that this took longer than 1…

Yeah maybe "expert engineer" is the wrong framing and it should be "oracle engineer" instead - you're right that we're not accounting for context switching (which, to be fair, is not really productive right?)

However ultimately the meaning isn't the absolute number but rather the relative difference (e.g. from PR to PR, or from team to team) - that's why we show industry benchmarks and make it easy to compare across teams!

Re: Show HN: Weave - actually measure engineering productivity

#22

Post by adchurch » Wed, Nov 20, 2024, 9:59 PM UTC

henning wrote: post #6

adchurch wrote: post #3

Our metric is approximately "hours of work for an expert engineer." Here are some example open source PRs and their output metrics calculated by our algorithm: https://github.com/PostHog/posthog/pull/25056 : 15.266 (Adds backend, frontend, and tests for a new feature) https://github.com/microsoft/vscode/pull/222315 : 8.401 (Refactors code to use a new service and adds new tests) https://github.com/facebook/react/pull…

That assumes all or almost all the work is writing the code, with no time allotted to actually using the app with that code written, benchmarking or other measurements, research about possible alternatives, etc.

Not at all! The algorithm is calibrated with real human effort. So find/replacing something 1000 times will have nowhere near the same value as adding 1000 lines of new code. And given 1000 lines of new code, you'll get the same value for implementing the same functionality in 100 lines instead.

What we don't capture is any product or communication overhead - however our platform has other metrics which can help find if these are causing inefficiencies :)

Re: Show HN: Weave - actually measure engineering productivity

#23

Post by id00 » Wed, Nov 20, 2024, 10:02 PM UTC

Let me just ignore my natural distain to the whole thing (as a engineer and a manager)

> We’ve developed a custom model that analyzes code and its impact directly...

This is a bold claim all things considering. Don't you need to fine tune this model for every customer as their business metrics likely vastly different? How do you measure the impact of refactoing? What about regressions or design mistakes that surface themselves after months or even years?

Re: Show HN: Weave - actually measure engineering productivity

#24

Post by adchurch » Wed, Nov 20, 2024, 10:03 PM UTC

henning wrote: post #18

adchurch wrote: post #17

Earlier quoted context omitted.

This metric approximately measures functionality added, so the way to game it would be to add lots more functionality. This metric has no opinion on the nature of that functionality (i.e. it's not evaluating product decisions). So it doesn't tell the whole story, but it tells a much more accurate story than LOC or whatever other metrics people are using currently!

You are describing a feature factory where people ship features just to ship features. Shitloads of code gets shipped and yet there's no real forward momentum because none of it has any impact. It is a bad place to be.

Wouldn't this be considered a very high performing engineering team with a very poor product team? We're only evaluating engineering - not the product decisions behind that engineering.

Re: Show HN: Weave - actually measure engineering productivity

#25

Post by adchurch » Wed, Nov 20, 2024, 10:11 PM UTC

rkagerer wrote: post #7

How did you come up with those magic correlation numbers? Is this generally just sniffing surface quality and quantity of written code, or is consideration given to how architecturally sound the system is built, whether the features introduced and their implementations make sense, how that power is exposed to users and whether the UI is approachable and efficient, user-feedback resulting from the effort, long-term su…

> How did you come up with those magic correlation numbers?

Evaluated on a proprietary data set of manually labelled PRs

> Is this generally just sniffing surface quality and quantity of written code...

Somewhere in between the two :) a PR with a poorly and quickly implemented login will have a lower output score than a PR with a robust, well-designed and tested login, simply because the latter is more effort. But there isn't (yet!) a metric to quantify the relative quality. So our metric doesn't tell the full story, but it gives more info than would have previously been available.

Re: Show HN: Weave - actually measure engineering productivity

#26

Post by adchurch » Wed, Nov 20, 2024, 10:13 PM UTC

karmajunkie wrote: post #8

adambeecee wrote: post #2

Hey HN! I'm one of the co-founders of Weave, and I wanted to jump in here to share a bit more. Building this has been a wild ride. The challenge of measuring engineering output in a way that’s fair and useful is something we’ve thought deeply about—especially because so many of the existing metrics feel fundamentally broken. The 0.94 correlation is based on rigorous validation with several teams (happy to dive into t…

I'd like to add that you need way more information on the landing page before I'm going to do much more than let you have my email address (if that.) Right now its a black box that takes in data(?) and spits out... something?

Check out our main landing page: https://workweave.dev/

Let me know if you have any questions that aren't answered there!

Re: Show HN: Weave - actually measure engineering productivity

#27

Post by adchurch » Wed, Nov 20, 2024, 10:17 PM UTC

sam_lowry_ wrote: post #12

vasco wrote: post #9

Earlier quoted context omitted.

And that's why runners don't track their race times.

Not all runners are competitive runners. I tack my time as a measure of progress among many other measurements, but my objective is to feel better and live a saner life. Metrics are harder on software engineers, the good ones delete code, the best ones make sure useless code never gets written in the first place. How do you measure that?

> Metrics are harder on software engineers

This is exactly the problem we're hoping to solve :')

> the good ones delete code

Our algorithm gives lots of credit for deleting old code!

> the best ones make sure useless code never gets written in the first place

This we don't really capture yet - but our hope is to continue along and give insights into this vital work as well down the line.

Re: Show HN: Weave - actually measure engineering productivity

#28

Post by adchurch » Wed, Nov 20, 2024, 10:22 PM UTC

senko wrote: post #11

"Hello Jane, please have a seat. We need to talk about your productivity. Yes, I know you helped the team through a crunch and delivered the new feature, which works flawlessly and is loved by our users. And our balance sheet is much healthier after you found that optimization that saves us $1mm/year. We also appreciate that younger teammates look to you for guidance and learn a lot from you. But you see, the AI scor…

LOL - I shudder at the idea of a manager making HR decisions based solely on this one metric!

To be clear we're not claiming this is 1 number to holistically evaluate an entire engineer. Rather we're giving a much more accurate picture of output, which most orgs are already measuring (with terrible accuracy). It should be an important part of the picture but certainly not the whole story!

And fwiw I think the scores are pretty transparent - in the platform, you can drill down into any number and see the actual PRs and their output measurements. Of course the underlying model is more complex but unfortunately simpler models are not sufficient to capture the way engineering output works.

Re: Show HN: Weave - actually measure engineering productivity

#29

Post by adchurch » Wed, Nov 20, 2024, 10:29 PM UTC

id00 wrote: post #23

Let me just ignore my natural distain to the whole thing (as a engineer and a manager) > We’ve developed a custom model that analyzes code and its impact directly... This is a bold claim all things considering. Don't you need to fine tune this model for every customer as their business metrics likely vastly different? How do you measure the impact of refactoing? What about regressions or design mistakes that surface…

Different companies will have different outputs just by the nature of their stage & situation, but the numbers are still relatively comparable (e.g. across different teams).

> How do you measure the impact of refactoing?

The metric def gives credit for refactoring

> What about regressions or design mistakes that surface themselves after months or even years?

Not captured (part of why it's only an important part of the story, not the whole story :))

Re: Show HN: Weave - actually measure engineering productivity

#30

Post by adchurch » Wed, Nov 20, 2024, 10:31 PM UTC

id00 wrote: post #23

Let me just ignore my natural distain to the whole thing (as a engineer and a manager) > We’ve developed a custom model that analyzes code and its impact directly... This is a bold claim all things considering. Don't you need to fine tune this model for every customer as their business metrics likely vastly different? How do you measure the impact of refactoing? What about regressions or design mistakes that surface…

> Let me just ignore my natural distain to the whole thing (as a engineer and a manager)

I totally get this - that's how I felt initially, but I was shocked to find that the vast majority of orgs are using bad metrics like LOC or commit counts anyway. Our belief is that replacing those with something much more accurate can help the entire industry.