Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We are changing our developer productivity experiment design

metr.org

Post reply on HN
Print view View original

21–30 of 62 posts

Re: We are changing our developer productivity experiment design

#21

Post by keeda »

> When surveyed, 30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI.

In fact, one of the developers in the original study later revealed on Twitter that he had already done exactly that during the study, i.e. filtered out tasks he prefered not to do without AI: https://xcancel.com/ruben_bloom/status/1943536052037390531

While this was only one developer (that we know of), given the N was 16 and he seems to have been one of the more AI-experienced devs, this could have had a non-trivial effect on the results.

The original study gets a lot of air-time from AI naysayers, let's see how much this follow-up gets ;-)

Re: We are changing our developer productivity experiment design

#22

Post by falcor84 »

daxfohl wrote: post #15

"I don't want to do this without AI" sounds like we're already well into the brain atrophy stage of this. Now what? (I'd think about it myself but....)

I'm pretty sure that this was exactly the response to the first generation of devs who insisted on coding with a terminal instead of submitting punch cards like "real programmers".

Re: We are changing our developer productivity experiment design

#23

Post by smohare »

simonw wrote: post #14
sgillen wrote: post #6

This is very interesting because I see a lot of AI detractors point to the original study as proof that AI is overhyped and nothing to worry about. In this new study the findings are essentially reversed (20% slowdown to 20% speedup).

AI detractors loved that previous study so much. It seems to have been brought up in the majority of conversations about AI productivity over the past six months. (Notable to me was how few other studies they cited, which I think is because studies showing AI productivity loss are quite uncommon.)

Or maybe there’s just not that many good studies, period?

A lot of them barely rise above the level of collected anecdote, nor explore long term or more elusive factors (such as cross-system entropy). They’re also targeting an area that is fairly difficult to measure and control for.

Re: We are changing our developer productivity experiment design

#24

Post by falcor84 »

camgunz wrote: post #9

Unless this measures the entire SDLC longitudinally (like say, over a year) I'm not interested. I too can tell Claude Code to do things all day every day, but unless we have data on the defect rate it doesn't matter at all.

Do any of those companies collect and share data on their defect rates to give you a baseline to compare against?

Re: We are changing our developer productivity experiment design

#25

Post by sjaiisba »

keeda wrote: post #21

> When surveyed, 30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI. In fact, one of the developers in the original study later revealed on Twitter that he had already done exactly that during the study, i.e. filtered out tasks he prefered not to do w…

> 3. Regarding me specifically, I work on the LessWrong codebase which is technically open-source. I feel like calling myself an "open-source developer" has the wrong connotations, and makes it more sound like I contribute to a highly-used Python library or something as an upper-tier developer which I'm not

That’s very interesting! This kinda matches what I see at work:

- low performers love it. it really does make them output more (which includes bugs, etc. it’s causing some contention that’s yet to be resolved)

- some high performers love it. these were guys who are more into greenfield stuff and ok with 90% good. very smart, but just not interested in anything outside of going fast

- everyone else seems to be finding use out of it, but reviews are painful

Re: We are changing our developer productivity experiment design

#26

Post by fxwin »

sgillen wrote: post #6

This is very interesting because I see a lot of AI detractors point to the original study as proof that AI is overhyped and nothing to worry about. In this new study the findings are essentially reversed (20% slowdown to 20% speedup).

fwiw i think the interesting part about the original study wasn't so much the slowdowm part, but the discrepancy between perceived and measured speedup/slowdown (which is the part i used to bring up frequently when talking to other devs)

Re: We are changing our developer productivity experiment design

#28

Post by tonymet »

> "AI tools lead to worse productivity"

> The subjects are using ChatGPT 2.5 and copy-pasting code.

The reason AI hype seems to be so bipolar is that "AI" isn't one thing. Hundreds of models, dozens of tools. And to get something done well, a seasoned engineer needs to master half a dozen at a time.

Re: We are changing our developer productivity experiment design

#29

Post by Wowfunhappy »

sjaiisba wrote: post #20
atleastoptimal wrote: post #12

It's kind of funny that METR is known primarily for both the most bearish study on AI progress (the original 20% slowdown one), and the most bullish one on AI progress (the long-task horizon study showing exponential increase in duration of tasks AI models can accomplish with respect to date of release). In either case, it seems people ended up bolstering their preexisting views on AI based on whichever study most af…

> Anthropic claims they solely use agents to code and don't modify any code manually. Have you used CC? It shows. They did not make their fortune off this, and it’s at least lost me a customer because of how sloppy it is. The model is good, and it’s why they have to gate access to it. I’d much rather use a different harness. I do think you’re on to something though. As societal wealth further concentrates among the f…

...uh, I think Claude Code is great, actually. A lot of that is indeed just the strength of the underlying model, but the local client is great too. Plan mode, checkpoints, subagents... I've been using Claude Code for a year now, and I feel like Anthropic has steadily been eliminating pain points.

It's certainly a lot better than the Gemini cli!

Re: We are changing our developer productivity experiment design

#30

Post by logicprog »

roxolotl wrote: post #17
ej88 wrote: post #2

Really interesting updates to their 2025 experiment. Repeat devs from the original experiment went from 0-40% slowdown to now -10-40% speedup - and METR estimates this as a 'lower-bound' more devs saying they dont even want to do 50% of their work without AI, even for 50/hr 30-50% of devs decided not to submit certain tasks without AI, missing the tasks with the highest uplift it also seems like there is a skill gap…

The finding of the first study was people cannot judge their performance with these tools. So I don’t think the lack of individuals not willing to work without them is indicative of productivity improvements. I think it’s indicative of them being enjoyable to use.

It was claimed to find that, but I don't think it did. It compared developers' beliefs about average speed up across tasks, measured by asking them once at the end, compared to the average comparative speed measured per task and then averaged. That's measuring two different things, and all kinds of things could mass up developers' fuzzy recollection of the gestalt of several tasks (such as recency bias and question/study framing) that wouldn't effect it if you asked them right after; moreover, when tasks were broken down by task type, the speed up/slow down results actually matched developers' qualitative comments.
Post reply on HN