Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

221–230 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#221

Post by i_love_retros »

noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

The problem is I then have to also figure out the code it wrote to be able to complete the final 20%. I have no momentum and am starting from almost scratch mentally.

Re: Measuring the impact of AI on experienced open-source developer productivity

#222

Post by Uehreka »

fwip wrote: post #144
Uehreka wrote: post #109

Earlier quoted context omitted.

Pardon my caps, but WHO CARES about acquisitions?! You’ve been given a dubiously capable genie that can write code without you having to do it! If this thing can build first drafts of those side projects you always think about and never get around to, that in and of itself is useful! If it can do the yak-shaving required to set up those e2e tests you know you should have but never have time for it is useful! Have it…

Unfortunately, HN is YC-backed, and attracts these types by design.

I mean sure, but HN/YC’s founder was always going on about the kinship between “Hackers and Painters” (or at least he used to). It hasn’t always been like this, and definitely doesn’t have to be. We can and should aspire to better.

Re: Measuring the impact of AI on experienced open-source developer productivity

#223

Post by luxpir »

smokel wrote: post #65
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I notice that some people have become more productive thanks to AI tools, while others are not. My working hypothesis is that people who are fast at scanning lots of text (or code for that matter) have a serious advantage. Being able to dismiss unhelpful suggestions quickly and then iterating to get to helpful assistance is key. Being fast at scanning code correlates with seniority, but there are also senior develope…

Just to thank you for that point. I think it's likely more true than most of us realise. That and maybe the ability to mentally scaffold or outline a system or solution ahead of time.

Re: Measuring the impact of AI on experienced open-source developer productivity

#224

Post by robwwilliams »

grey-area wrote: post #98
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wro…

Or a sampling artifact. 4 vs 12 does seem significant within a study, but consider a set of N such studies.

I assume that many large companies have tested efficiency gains and losses of there programmers much more extensively than the authors of this tiny study.

A survey of companies and their evaluation and conclusions would carry more weight—-excluding companies selling AI products, of course.

Re: Measuring the impact of AI on experienced open-source developer productivity

#225

Post by AvAn12 »

sarchertech wrote: post #129
AvAn12 wrote: post #118

N = 16 developers. Is this enough to draw any meaningful conclusions?

That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection. If you are looking for a 0.1% increase in productivity, then 16 is too small.

“A quarter of the participants saw increased performance, 3/4 saw reduced performance.” So I think any conclusions drawn on these 16 people doesn’t signify much one way or the other. Cool paper but how is this anything other than a null finding?

Re: Measuring the impact of AI on experienced open-source developer productivity

#226

Post by Terr_ »

bryanrasmussen wrote: post #196
pdabbadabba wrote: post #184

Earlier quoted context omitted.

Maybe it's convenient. But isn't it also just a fact that some of the models available today are better than the ones available five months ago?

sure, but after having spent some time trying to get anything useful - programmatically - out of previous models and not getting anything once a new one is announced how much time should one spend. Sure you may end up missing out on a good thing and then having to come late to the party, but coming early to the party too many times and the beer is watered down and the food has grubs is apt to make you cynical the nex…

Plus it's not even possible to miss the metaphorical party: If it gets going, it will be quite obvious long before it peaks.

(Unless one believes the most grandiose prophecies of a technological-singularity apocalypse, that is.)

Re: Measuring the impact of AI on experienced open-source developer productivity

#227

Post by gojomo »

narush wrote: post #139
gojomo wrote: post #132

Earlier quoted context omitted.

Did each developer do a large enough mix of AI/non-AI tasks, in varying orders, that you have any hints in your data whether the "AI penalty" grew or shrunk over time?

You can see this analysis in the factor analysis of "Below-average use of AI tools" (C.2.7) in the paper [1], which we mark as an unclear effect. TLDR: over the first 8 issues, developers do not appear to get majorly less slowed down. [1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

Thanks, that's great!

But: if all developers did 136 AI-assisted issues, why only analyze excluding the 1st 8, rather than, say, the first 68 (half)?

Re: Measuring the impact of AI on experienced open-source developer productivity

#228

Post by Filligree »

Ntrails wrote: post #204
cwillu wrote: post #180

Earlier quoted context omitted.

The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed. Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true…

Some of it is just that (probably different) people said the same damn things 6 months ago. "No, the 2.8 release is the first good one. It massively improves workflows" Then, 6 months later, the study comes out. "Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add" At some point, you roll your eyes and assume it is just snake oil sales

Or you accept that different people have different skill levels, workflows and goals, and therefore the AIs reach usability at different times.

Re: Measuring the impact of AI on experienced open-source developer productivity

#229

Post by lurking_swe »

furyofantares wrote: post #197
bluefirebrand wrote: post #191

Earlier quoted context omitted.

Greenfield is still such a tiny percentage of all software work going on in the world though :/

I agree, that's fair. I think a lot of people are playing around with AI on side projects and making some bad extrapolations from their initial experiences. It'll also apply to isolated-enough features, which is still a small amount of someone's work (not often something you'd work on for a full month straight), but more people will have experience with this.

greenfield development is also the “easiest” and most fun part of software development. As the famous saying goes, the last 10% of the project takes 90% of the time lol.

I’ve also noticed that, generally, nobody likes maintaining old systems.

so where does this leave us as software engineers? Should I be excited that it’s easy to spin up a bunch of code that I don’t deeply understand at the beginning of my project, while removing the fun parts of the project?

I’m still grappling with what this means for our industry in 5-10 years…

Re: Measuring the impact of AI on experienced open-source developer productivity

#230

Post by steveklabnik »

Ntrails wrote: post #204
cwillu wrote: post #180

Earlier quoted context omitted.

The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed. Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true…

Some of it is just that (probably different) people said the same damn things 6 months ago. "No, the 2.8 release is the first good one. It massively improves workflows" Then, 6 months later, the study comes out. "Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add" At some point, you roll your eyes and assume it is just snake oil sales

There’s a lot of confounding factors here. For example, you could point to any of these things in the last ~8 months as being significant changes:

* the release of agentic workflow tools

* the release of MCPs

* the release of new models, Claude 4 and Gemini 2.5 in particular

* subagents

* asynchronous agents

All or any of these could have made for a big or small impact. For example, I’m big on agentic tools, skeptical of MCPs, and don’t think we yet understand subagents. That’s different from those who, for example, think MCPs are the future.

> At some point, you roll your eyes and assume it is just snake oil sales

No, you have to realize you’re talking to a population of people, and not necessarily the same person. Opinions are going to vary, they’re not literally the same person each time.

There are surely snake oil salesman, but you can’t buy anything from me.

Post reply on HN