Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

231–240 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#231

Post by card_zero »

atiedebee wrote: post #218
grey-area wrote: post #98

Earlier quoted context omitted.

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wro…

Let me bring you a third (not necessarily true) interpretation: The developer who has experience using cursor saw a productivity increase not because he became better at using cursor, but because he became worse at not using it.

Or, one person in 16 has a particular personality, inclined to LLM dependence.

Re: Measuring the impact of AI on experienced open-source developer productivity

#233

Post by evanelias »

pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific:

What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes?

If that was the case, wouldn't we expect developers to have an overly positive perception of AI because they're literally becoming addicted to it?

Re: Measuring the impact of AI on experienced open-source developer productivity

#234

Post by IshKebab »

narush wrote: post #96
IshKebab wrote: post #11

I wonder if the discrepancy is that it felt like it was taking less time because they were having to do less thinking which feels like it is easier and hence faster. Even so... I still would be really surprised if there wasn't some systematic error here skewing the results, like the developers deliberately picked "easy" tasks that they already knew how to do, so implementing them themselves was particularly fast. See…

> which feels like it is easier and hence faster. We explore this factor in section (C.2.5) - "Trading speed for ease" - in the paper [1]. It's labeled as a factor with an unclear effect, some developers seem to think so, and others don't! > like the developers deliberately picked "easy" tasks that they already knew how to do We explore this factor in (C.2.2) - "Unrepresentative task distribution." I think the effect…

Sounds like you've thought of everything!

Re: Measuring the impact of AI on experienced open-source developer productivity

#235

Post by steveklabnik »

foobarqux wrote: post #212
simonw wrote: post #171

Earlier quoted context omitted.

The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.

That's not the argument being made though, which is that it does "work" now and implying that actually it didn't quite work before; except that that is the same thing the same people say for every model release, including at the time or release of the previous one, which is now acknowledged to be seriously flawed; and including the future one, at which time the current models will similarly be acknowledged to be, not…

> That's not the argument being made though, which is that it does "work" now and implying that actually it didn't quite work before

Right.

> except that that is the same thing the same people say for every model release,

I did not say that, no.

I am sure you can find someone who is in a Groundhog Day about this, but it’s just simpler than that: as tools improve, more people find them useful than before. You’re not talking to the same people, you are talking to new people each time who now have had their threshold crossed.

Re: Measuring the impact of AI on experienced open-source developer productivity

#236

Post by PessimalDecimal »

gmaster1440 wrote: post #136

What if the slowdown isn't a bug but a feature? What if AI tools are forcing developers to think more carefully about their code, making them slower but potentially producing better results? AFAIK the study measured speed, not quality, maintainability, or correctness. The developers might feel more productive because they're engaging with their code at a higher level of abstraction, even if it takes longer. This woul…

In my experience, LLMs are not causing people to think more carefully about their code.

Re: Measuring the impact of AI on experienced open-source developer productivity

#237

Post by Uehreka »

steveklabnik wrote: post #215
mattmanser wrote: post #202

Earlier quoted context omitted.

Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model c…

Yes. In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan. As with anything, your miles may vary: I’m not here to tell anyone that thinks they still suck that their experience is invalid, but to me it’s been a pretty big swing.

> In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan.

Same. For me the turning point was VS Code’s Copilot Agent mode in April. That changed everything about how I work, though it had a lot of drawbacks due to its glitches (many of these were fixed within 6 or so weeks).

When Claude Sonnet 4 came out in May, I could immediately tell it was a step-function increase in capability. It was the first time an AI, faced with ambiguous and complicated situations, would be willing to answer a question with a definitive and confident “No”.

After a few weeks, it became clear that VS Code’s interface and usage limits were becoming the bottleneck. I went to my boss, bullet points in hand, and easily got approval for the Claude Max $200 plan. Boom, another step-function increase.

We’re living in an incredibly exciting time to be a skilled developer. I understand the need to stay skeptical and measure the real benefits, but I feel like a lot of people are getting caught up in the culture war aspect and are missing out on something truly wonderful.

Re: Measuring the impact of AI on experienced open-source developer productivity

#238

Post by Filligree »

bluefirebrand wrote: post #191
furyofantares wrote: post #178

Earlier quoted context omitted.

I'm specifically talking about greenfield work. I do a lot of game prototypes, it definitely does that at the very beginning.

Greenfield is still such a tiny percentage of all software work going on in the world though :/

It’s a tiny percentage of software work because the programming is slow, and setting up new projects is even slower.

It’s been a majority of my projects for the past two months. Not because work changed, but because I’ve written a dozen tiny, personalised tools that I wouldn’t have written at all if I didn’t have Claude to do it.

Most of them were completed in less than an hour, to give you an idea of the size. Though it would have easily been a day on my own.

Re: Measuring the impact of AI on experienced open-source developer productivity

#239

Post by EarthLaunch »

evanelias wrote: post #233
pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

> The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con

https://softwarecrisis.dev/letters/llmentalist/

Plus there's a gambling mechanic: Push the button, sometimes get things for free.

Re: Measuring the impact of AI on experienced open-source developer productivity

#240

Post by Filligree »

smokel wrote: post #65
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I notice that some people have become more productive thanks to AI tools, while others are not. My working hypothesis is that people who are fast at scanning lots of text (or code for that matter) have a serious advantage. Being able to dismiss unhelpful suggestions quickly and then iterating to get to helpful assistance is key. Being fast at scanning code correlates with seniority, but there are also senior develope…

An interesting point. I wonder how much my decades-old habit of watching subtitled anime helps there—it’s definitely made me dramatically faster at scanning text.
Post reply on HN