Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

491–500 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#491

Post by robenkleene »

fiddlerwoaroof wrote: post #368
pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

I think just about every developer hack turns out this way: static vs dynamic types; keyboard shortcuts vs mice; etc. But I think it’s also possible to over-interpret these findings: using the tools that make your work enjoyable has important second-order effects even if they aren’t the productivity silver bullet everyone claims they are.

This same conclusion has been actually observed practically verbatim about the keyboard vs. mouse https://www.asktog.com/TOI/toi06KeyboardVMouse1.html from the post:

- Test subjects consistently report that keyboarding is faster than mousing.

- The stopwatch consistently proves mousing is faster than keyboarding.

Re: Measuring the impact of AI on experienced open-source developer productivity

#492

Post by fiddlerwoaroof »

robenkleene wrote: post #491
fiddlerwoaroof wrote: post #368

Earlier quoted context omitted.

I think just about every developer hack turns out this way: static vs dynamic types; keyboard shortcuts vs mice; etc. But I think it’s also possible to over-interpret these findings: using the tools that make your work enjoyable has important second-order effects even if they aren’t the productivity silver bullet everyone claims they are.

This same conclusion has been actually observed practically verbatim about the keyboard vs. mouse https://www.asktog.com/TOI/toi06KeyboardVMouse1.html from the post: - Test subjects consistently report that keyboarding is faster than mousing. - The stopwatch consistently proves mousing is faster than keyboarding.

Yeah, that was one of the pieces of research I was thinking about. Another is this review of the evidence, such as it is, for static types: https://danluu.com/empirical-pl/

So far, Fred Brook’s “No Silver Bullet” remains undefeated.

Re: Measuring the impact of AI on experienced open-source developer productivity

#493

Post by Kon5ole »

ivanovm wrote: post #309
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

>It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user.

I have the opposite impression! I find it's hard to think of any other tech product where users expect to master it with no training at all. I think people get tricked into believing they need no training because the tool uses natural language as the UI.

You learn how to use a spreadsheet or a word processor, how to drive a car, sail a boat, play a guitar. In the 90s there were courses that spent hours teaching users how to work a mouse and keyboard!

Of course you need to learn how to use a coding assistant as well, it just makes sense.

There has already been a million words written about how to use LLMs from people who don't really know how to use LLM's. Everyone is learning, there is a boom, you can make a fortune selling knowledge about LLM's whether you have that knowledge or not.

Re: Measuring the impact of AI on experienced open-source developer productivity

#494

Post by sarlalian »

ivanovm wrote: post #309
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

There are plenty of examples of other tech where "you're just not using it right" is perfectly acceptable and many people find that it provides a high level of value. Rust and Vim being two that come immediately to mind. Both have sharp edges and steep learning curves, yet for some population are wildly popular, yet not right for everyone.

It's also possible for the user to be not using it right and that not be a value judgement on the user. We all suck at using new tools, that's part of learning.

Re: Measuring the impact of AI on experienced open-source developer productivity

#495

Post by ColonelPhantom »

eightysixfour wrote: post #270
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I have been teaching people at my company how to use AI code tools, the learning curve is way worse for developers and I have had to come up with some exercises to try and breakthrough the curve. Some seemingly can’t get it. The short version is that devs want to give instructions instead of ask for what outcome they want. When it doesn’t follow the instructions, they double down by being more precise, the worst thin…

Why is giving precise instructions bad? I would expect LLMs to be pretty good at following instructions after three years of training them that way. Plus, if the instructions are precise enough and therefore each step is simple enough, I would expect everything it needs to do to be 'in-distribution'.

Re: Measuring the impact of AI on experienced open-source developer productivity

#497

Post by ieie3366 »

lloeki wrote: post #421
ieie3366 wrote: post #208

LLMs are godtier if you know what you’re doing, and prompt them with ”do X”, where x is a SELF-CONTAINED change you would manually know how to implement For example, today I asked claude to implement per-user rate-limiting into my nestjs service, then iterated by asking implementing specific unit tests and some refactoring. It one-shot everything. I would say 90% time savings. Unskilled people ask them ”i have giant…

I tried exactly that, several times, over and over. Except on "hello world" situations (which I guess is a solid part of the corpus LLMs are trained with) these tools were consistently slower. Last time was an area where several files were subtly different in a section that essentially does about the same thing, and needed to be aligned and made consistent†. Time to - begrudgingly - do it manually: 5min Time to come…

Please specify the model. If you’re using chatgpt’s default model as is common, it is complete useless slop

Re: Measuring the impact of AI on experienced open-source developer productivity

#499

Post by GregDavidson »

AI used to refer to the extensive range of techniques of the field of Artificial Intelligence. Now it refers to LLMs and maybe other multi-layer networks trained on vast datasets. LLMs are great for some tasks and are also great as parts of hybrid systems like the IBM Watson Jepardy system. There's much more to Artificial Intelligence, e.g. https://en.m.wikipedia.org/wiki/Knowledge_representation_and... et al.

Re: Measuring the impact of AI on experienced open-source developer productivity

#500

Post by smcleod »

Claude 4 Opus when provided with the full paper, blog post and asked to perform a critical review:

The methodology contains several fundamental flaws that likely explain this anomalous result.

Most critically, the study examines a highly specific scenario - expert developers working on codebases they've contributed to for years (averaging 1,500 commits over 5 years) - which creates a ceiling effect where AI has minimal room to provide value.

The 30-minute Cursor training for developers, 56% of whom had never used the tool before, is woefully inadequate for learning effective AI pair programming techniques. With only 16 participants and a non-blinded design where developers knew their condition and were paid $150/hour, the study lacks both statistical power and ecological validity.

The restriction to a single tool configuration (Cursor Pro with specific Claude models) and acknowledgment that the tool doesn't optimise token sampling or prompting strategies further limits the findings' applicability to the broader question of AI's impact on developer productivity.

- Concerningly small sample size: Only 16 developers across 246 tasks provides insufficient statistical power for broad generalisations about AI's impact on millions of developers

- Inadequate AI training: 30-minute basic Cursor tutorial for developers where 56% had never used the tool - completely insufficient for developing effective AI collaboration skills

- Selection bias towards ceiling effects: Developers averaged 5 years and 1,500 commits on their repositories, creating an expertise level where AI assistance has minimal value-add potential

- Single tool learning: Study restricted to education Cursor Pro with specific Claude models, not representative of the diverse AI tooling ecosystem (Windsurf, Cline, Roo Code etc.)

- Artificial task constraints: Tasks were acknowledged as "shorter than average" and broken into ≤2 hour chunks, not representative of real development work

- No experimental blinding: Developers knew their condition and were being observed/recorded, potentially affecting natural work patterns

- Suboptimal AI usage: Study acknowledges Cursor doesn't sample sufficient tokens and developers reported overusing AI due to experimental conditions

- Narrow context: All repositories were large (1.1M LoC average), mature (10 years old), with high quality standards - a specific niche not representative of most development

- Self-reported metrics: Time tracking was self-reported with only 29% verified through screen recordings, introducing measurement bias

- Expertise mismatch: Focusing on experts contradicts established findings that AI tools provide greater benefits to less experienced developers

- Tool proficiency confound: No control for varying levels of AI tool proficiency or different prompting strategies between participants

- Limited generalisability: The specific combination of expert developers + familiar codebases + large repositories + short tasks creates an artificial scenario unlike typical development workflows

Post reply on HN