Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Weave - actually measure engineering productivity

app.workweave.ai

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Show HN: Weave - actually measure engineering productivity

#11

Post by senko »

"Hello Jane, please have a seat. We need to talk about your productivity. Yes, I know you helped the team through a crunch and delivered the new feature, which works flawlessly and is loved by our users. And our balance sheet is much healthier after you found that optimization that saves us $1mm/year. We also appreciate that younger teammates look to you for guidance and learn a lot from you.

But you see, the AI scored your productivity at 47%, barely "meets expectations", while we expect everyone to score at least 72%, "exceeds expectations". How is that calculated? The AI is a state of the art proprietary model, I don't know the details...

Anyways, we've got to design a Personal Improvement Plan for you. Here's what our AI recommends. We'll start with the TPS reports..."

Re: Show HN: Weave - actually measure engineering productivity

#12

Post by sam_lowry_ »

vasco wrote: post #9
henning wrote: post #5

As soon as people know how the metric is calculated, they will game that metric and it will cease to be useful.

And that's why runners don't track their race times.

Not all runners are competitive runners. I tack my time as a measure of progress among many other measurements, but my objective is to feel better and live a saner life.

Metrics are harder on software engineers, the good ones delete code, the best ones make sure useless code never gets written in the first place. How do you measure that?

Re: Show HN: Weave - actually measure engineering productivity

#13

Post by ryandvm »

vasco wrote: post #9
henning wrote: post #5

As soon as people know how the metric is calculated, they will game that metric and it will cease to be useful.

And that's why runners don't track their race times.

If the metric is COINCIDENT with the goal, then it's not a problem. You can't get better at winning races without improving your race time (well... you can degrade everyone else's times I suppose).

But if the metric is a PROXY for the goal, then the metric becomes the objective (not the actual goal).

In this case, whatever this AI is measuring is going to be exactly what every dev drops into their Github Copilot prompt instructions.

Re: Show HN: Weave - actually measure engineering productivity

#14

Post by jumploops »

adchurch wrote: post #3

Our metric is approximately "hours of work for an expert engineer." Here are some example open source PRs and their output metrics calculated by our algorithm: https://github.com/PostHog/posthog/pull/25056 : 15.266 (Adds backend, frontend, and tests for a new feature) https://github.com/microsoft/vscode/pull/222315 : 8.401 (Refactors code to use a new service and adds new tests) https://github.com/facebook/react/pull…

Curious how these numbers correlate to the estimates of the engineers behind the PRs?

For example, the first PR is correlated with ~15 "hours of work for an expert engineer"

Looking at the PR, it was opened on Sept 18th and merged on Oct 2nd. That's two weeks, or 10 working days, later.

Between the initial code, the follow up PR feedback, and merging with upstream (8 times), I would wager that this took longer than 15 hours of work on the part of the author.

It doesn't _really_ matter, as long as the metrics are proportional, but it may be better to refer to them as isolated complexity hours, as context-switching doesn't seem to be properly accounted for.

Re: Show HN: Weave - actually measure engineering productivity

#15

Post by dragonwriter »

ryandvm wrote: post #13
vasco wrote: post #9

Earlier quoted context omitted.

And that's why runners don't track their race times.

If the metric is COINCIDENT with the goal, then it's not a problem. You can't get better at winning races without improving your race time (well... you can degrade everyone else's times I suppose). But if the metric is a PROXY for the goal, then the metric becomes the objective (not the actual goal). In this case, whatever this AI is measuring is going to be exactly what every dev drops into their Github Copilot prom…

Making a metric an objective is an effective if often very costly way of testing the hypothesis that it is coincident rather than a proxy (which in cases more complex than racing is frequently a matter of dispute.)

Re: Show HN: Weave - actually measure engineering productivity

#17

Post by adchurch »

henning wrote: post #5

As soon as people know how the metric is calculated, they will game that metric and it will cease to be useful.

This metric approximately measures functionality added, so the way to game it would be to add lots more functionality.

This metric has no opinion on the nature of that functionality (i.e. it's not evaluating product decisions). So it doesn't tell the whole story, but it tells a much more accurate story than LOC or whatever other metrics people are using currently!

Re: Show HN: Weave - actually measure engineering productivity

#18

Post by henning »

adchurch wrote: post #17
henning wrote: post #5

As soon as people know how the metric is calculated, they will game that metric and it will cease to be useful.

This metric approximately measures functionality added, so the way to game it would be to add lots more functionality. This metric has no opinion on the nature of that functionality (i.e. it's not evaluating product decisions). So it doesn't tell the whole story, but it tells a much more accurate story than LOC or whatever other metrics people are using currently!

You are describing a feature factory where people ship features just to ship features. Shitloads of code gets shipped and yet there's no real forward momentum because none of it has any impact. It is a bad place to be.

Re: Show HN: Weave - actually measure engineering productivity

#19

Post by adchurch »

mg57 wrote: post #16

Pretty dumb to think you can infer effort from the code itself. You make one "smart invocation" to a remote microservice and replace 1000 lines of code! The information for effort is not available at the code level - sorry to burst your bubble.

Our algorithm gives lots of credit to removing old lines of code!

But to your broader point, I think there certainly is information about effort at the code level. Consider for example these two PRs: https://github.com/PostHog/posthog/pull/23858 and https://github.com/microsoft/vscode/pull/209557. It's pretty easy to tell which one was more effort even if you don't know anything about the process for how they were implemented.

Do you have any shareable examples you want me to test out? Or of course you can try it yourself :)

Re: Show HN: Weave - actually measure engineering productivity

#20

Post by adchurch »

cssanchez wrote: post #4
adambeecee wrote: post #2

Hey HN! I'm one of the co-founders of Weave, and I wanted to jump in here to share a bit more. Building this has been a wild ride. The challenge of measuring engineering output in a way that’s fair and useful is something we’ve thought deeply about—especially because so many of the existing metrics feel fundamentally broken. The 0.94 correlation is based on rigorous validation with several teams (happy to dive into t…

Skeptic here. How can you validate the difference in effort from a startup where growth happens in explosive moments with many rewrites in between vs a refined enterprise codebase with incremental changes? Is it productive if I have tried many changes in branches and none of them made it to prod?

Startups will naturally have higher output than enterprises for this reason - we'll show people benchmarks accordingly.

> Is it productive if I have tried many changes in branches and none of them made it to prod?

Our metric measures displacement, not distance - under the assumption that the end state is the part that matters the most. It will notice if the resulting change has a higher cognitive load and evaluate it accordingly - but if there is no resulting change then ultimately there's no output to measure.

Post reply on HN