Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

161–170 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#161

Post by isoprophlex »

Ed Zitron was 100% right. The mask is off and the AI subprime crisis is coming. Reading TFA, it would be hilarious if the bubble burst AND it turns out there's actually no value to be had, at ANY price. I for one can't wait for this era of hype to end. We'll see.

you're addicted to the FEELING of productivity more than actual productivity. even knowing this, even seeing the data, even acknowledging the complete fuckery of it all, you're still gonna use me. i'm still gonna exist. you're all still gonna pretend this helps because the alternative is admitting you spent billions of dollars on spicy autocomplete.

Re: Measuring the impact of AI on experienced open-source developer productivity

#162

Post by sandinmyjoints »

Terr_ wrote: post #147
grey-area wrote: post #98

Earlier quoted context omitted.

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wro…

> people consistently predict and self-report in the wrong direction I recall an adage about work-estimation: As chunks get too big, people unconsciously substitute "how possible does the final outcome feel" with "how long will the work take to do." People asked "how long did it take" could be substituting something else, such as "how alone did I feel while working on it."

That’s an interesting adage. Any ideas of its source?

Re: Measuring the impact of AI on experienced open-source developer productivity

#163

Post by throwawayoldie »

mnky9800n wrote: post #159
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I feel like I get better at it as I use Claude code more because I both understand its strength and weaknesses and also understand what context it’s usually missing. Like today I was struggling to debug an issue and realised that Claude’s idea of a coordinate system was 90 degrees rotated from mine and thus it was getting confused because I was confusing it.

One of the major findings is that people's perception--that is, what it felt like--was incorrect.

Re: Measuring the impact of AI on experienced open-source developer productivity

#164

Post by keerthiko »

IME AI coding is excellent for one-off scripts, personal automation tooling (I iterate on a tool to scrape receipts and submit expenses for my specific needs) and generally stuff that can be run in environments where the creator and the end user are effectively the same (and only) entity.

Scaled up slightly, we use it to build plenty of internal tooling in our video content production pipeline (syncing between encoding tools and a status dashboard for our non-technical content team).

Using it for anything more than boilerplate code, well-defined but tedious refactors, or quickly demonstrating how to use an unfamiliar API in production code, before a human takes a full pass at everything is something I'm going to be wary of for a long time.

Re: Measuring the impact of AI on experienced open-source developer productivity

#165

Post by amirhirsch »

amirhirsch wrote: post #153
narush wrote: post #60

Earlier quoted context omitted.

Hey Simon -- thanks for the detailed read of the paper - I'm a big fan of your OS projects! Noting a few important points here: 1. Some prior studies that find speedup do so with developers that have similar (or less!) experience with the tools they use. In other words, the "steep learning curve" theory doesn't differentially explain our results vs. other results. 2. Prior to the study, 90+% of developers had reasona…

Figure 6 which breaks-down the time spent doing different tasks is very informative -- it suggest: 15% less active coding 5% less testing, 8% less research and reading 4% more idle time 20% more AI interaction time The 28% less coding/testing/research is why developers reported 20% less work. You might be spending 20% more time overall "working" while you are really idle 5% more time and feel like you've worked less…

i just realized the figure is showing the time breakdown as a percentage of total time, it would be more useful to show absolute time (hours) for those side-by-side comparisons since the implied hours would boost the AI bars height by 18%

Re: Measuring the impact of AI on experienced open-source developer productivity

#166

Post by biophysboy »

narush wrote: post #67
Jabrov wrote: post #2

Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.

Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues (7.5 with AI and 7.5 without AI) on average.

Did you compare the variance within individuals (due to treatment) to the variance between individuals (due to other stuff)?

Re: Measuring the impact of AI on experienced open-source developer productivity

#167

Post by Dilettante_ »

sandinmyjoints wrote: post #162
Terr_ wrote: post #147

Earlier quoted context omitted.

> people consistently predict and self-report in the wrong direction I recall an adage about work-estimation: As chunks get too big, people unconsciously substitute "how possible does the final outcome feel" with "how long will the work take to do." People asked "how long did it take" could be substituting something else, such as "how alone did I feel while working on it."

That’s an interesting adage. Any ideas of its source?

It might have been in Kahneman's "Thinking, Fast and Slow"

Re: Measuring the impact of AI on experienced open-source developer productivity

#168

Post by isoprophlex »

narush wrote: post #28

Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093

I'll just say that the methodology of the paper and the professionalism with which you are answering us here is top notch. Great work.

Re: Measuring the impact of AI on experienced open-source developer productivity

#169

Post by blibble »

steveklabnik wrote: post #151
grey-area wrote: post #98

Earlier quoted context omitted.

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wro…

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

> One thing that happened here is that they aren't using current LLMs

I've been hearing this for 2 years now

the previous model retroactively becomes total dogshit the moment a new one is released

convenient, isn't it?

Re: Measuring the impact of AI on experienced open-source developer productivity

#170

Post by mrwaffle »

My overall concern has to do with our developer ecosystem from the important points mentioned by simonw and narush. I've been concerned about this for years but AI reliance seems to be pouring jet fuel on the fire. Particularly troubling is the lack of understanding less-experienced devs will have over time. Does anyone have a counter-argument for this they can share on why this is a good thing?
Post reply on HN