> We pay developers $150/hr as compensation for their participation in the study. Can someone point me to these 300k/yr jobs?
These are not 300k/yr jobs.
Measuring the impact of AI on experienced open-source developer productivity
81–90 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#82Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
Hey I just wanted to say this is one of the better studies I've seen - not clickbaity, very forthright about what is being claimed, and presented in such an easy-to-digest format. Thanks so much for doing this.
Re: Measuring the impact of AI on experienced open-source developer productivity
#83Earlier quoted context omitted.
The issue is that it is slow and verbose, at least in its default configuration. The amount of reading is non trivial. There’s a reason most references are dense.
Those issues you can partly solve by changing the prompt to tell it to be concise and don't explain its code. But nothing will make them stick to the one API version I use.
> But nothing will make them stick to the one API version I use.
Models trained for tool use can do that. When I use Codex for some Rust stuff for example, it can grep from source files in the directory dependencies are stored, so looking up the current APIs is trivial for them. Same works for JavaScript and a bunch of other languages too, as long as it's accessible somewhere via the tools they have available.
Re: Measuring the impact of AI on experienced open-source developer productivity
#84> developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I feel like there are two challenges causing this. One is that it's difficult to get good data on how long the same person in the same context would have taken to do a task without AI vs with. The other is that it's tempting to time an AI with metrics like how long until the PR w…
Qualitatively, we don't see a drop in PR quality in between AI-allowed and AI-disallowed conditions in the study; the devs who participate are generally excellent, know their repositories standards super well, and aren't really into the 'get up a bad PR' vibe -- the median review time on the PRs in the study is about a minute.
Developers totally spend time totally differently, though, this is a great callout! On page 10 of the paper [1], you can see a breakdown of how developers spend time when they have AI vs. not - in general, when these devs have AI, they spend a smaller % of time writing code, and a larger % of time working with AI (which... makes sense).
[1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
Re: Measuring the impact of AI on experienced open-source developer productivity
#85Earlier quoted context omitted.
> My intuition here is that this study mainly demonstrated that the learning curve on AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learing curve. Definitely. Effective LLM usage is not as straightforward as people believe. Two big things I see a lot of developers do when they share chats: 1. Talk to the LLM like…
"But I can assure you that "pandas count unique values column 'Foo'" is just as effective an LLM prompt as "Using pandas, how do I get the count of unique values in the column named 'Foo'?"" How can you be so sure? Did you compare in a systematic way or read papers by people who did it? Now I surely get results giving the llm only snippets and keywords, but anything complex, I do notice differences the way I articula…
> How can you be so sure? Did you compare in a systematic way or read papers by people who did it?
No, but I didn't need to read scientific papers to figure how to use Google effectively, either. I'm just using a results-based analysis after a lot of LLM usage.
Re: Measuring the impact of AI on experienced open-source developer productivity
#86Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
Could you either release the dataset (raw but anonymized) for independent statistical évaluation or at least add the absolute times of each dev per task to the paper? I'm curious what the absolute times of each dev with/without AI was and whether the one guy with lots of Cursor experience was actually faster than the rest of just a slow typer getting a big boost out of llms
Also, cool work, very happy to see actually good evaluations instead of just vibes or observational stuies that don't account for the Hawthorne effect
Re: Measuring the impact of AI on experienced open-source developer productivity
#87This does not mention the open-source developer time wasted while reviewing vibe coded PRs
Yeah, I'll note that this study does _not_ capture the entire OS dev workflow -- you're totally right that reviewing PRs is a big portion of the time that many maintainers spend on their projects (and thanks to them for doing this [often hard] work). In the paper [1], we explore this factor in more detail -- see section (C.2.2) - Unrepresentative task distribution.
There's some existing lit about increased contributions to OS repositories after the introduction of AI -- I've also personally heard a fear anecdotes about an increase in the number of low-quality PRs from first time contributors, seemingly as a result of AI making it easier to get started -- ofc, the tradeoff is that making it easier to get started has pros to it too!
[1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
Re: Measuring the impact of AI on experienced open-source developer productivity
#88Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect.
You hit the nail on the head here.
I feel like I’ve seen a lot of people trying to make strong arguments that AI coding assistants aren’t useful. As someone who uses and enjoys AI coding assistants, I don’t find this research angle to be… uh… very grounded in reality?
Like, if you’re using these things, the fact that they are useful is pretty irrefutable. If one thinks there’s some sort of “productivity mirage” going on here, well OK, but to demonstrate that it might be better to start by acknowledging areas where they are useful, and show that your method explains the reality we’re seeing before using that method to show areas where we might be fooling ourselves.
I can maybe buy that AI might not be useful for certain kinds of tasks or contexts. But I keep pushing their boundaries and they keep surprising me with how capable they are, so it feels like it’ll be difficult to prove otherwise in a durable fashion.
Re: Measuring the impact of AI on experienced open-source developer productivity
#89Earlier quoted context omitted.
This is really disingenuous when you also say that OpenAI and Anthropic have provided you with access and compute credits (on https://metr.org/about ). Not all payment is cash. Compute credits is still by all means compensation.
Are you willing to be compensated with compute credits for your job? Such companies spit out "credits" all over the place in order to gain traction and enstablish themselves. I remember when cloud providers gave vps credits to startups like they were peanuts. To me, it really means absolutelly nothing.
I wouldn't do my job for $10, but if somehow someone did pay me $10 to do something, i wouldn't claim i wasn't compensated.
In-kind compensation is still compensation.
Re: Measuring the impact of AI on experienced open-source developer productivity
#90Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
Hey Simon -- thanks for the detailed read of the paper - I'm a big fan of your OS projects! Noting a few important points here: 1. Some prior studies that find speedup do so with developers that have similar (or less!) experience with the tools they use. In other words, the "steep learning curve" theory doesn't differentially explain our results vs. other results. 2. Prior to the study, 90+% of developers had reasona…
Really interesting paper, and thanks for the followon points.
The over-optimism is indeed a really important takeaway, and agreed that it's not tool-dependent.