AI by design can only repeat and recombine past material. Therefore actual invention is out.
HN moment
Measuring the impact of AI on experienced open-source developer productivity
121–130 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#122Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
Was any attention paid to whether the tickets being implemented with AI assistance were an appropriate use case for AI? If the instruction is just "implement this ticket with AI", then that's very realistic in that it's how management often tries to operate, but it's also likely to be quite suboptimal. There are ways to use AI that help a lot, and other ways that hurt more than it helps. If your developers had suffic…
The instructions given to developers was not just "implement with AI" - but rather that they could use AI if they deemed it would be helpful, but indeed did _not need to use AI if they didn't think it would be helpful_. In about ~16% of labeled screen recordings where developers were allowed to use AI, they choose to use no AI at all!
That being said, we can't rule out that the experiment drove them to use more AI than they would have outside of the experiment (in a way that made them less productive). You can see more in section "Experimentally driven overuse of AI (C.2.1)" [1]
[1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
Re: Measuring the impact of AI on experienced open-source developer productivity
#123Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
Hey I just wanted to say this is one of the better studies I've seen - not clickbaity, very forthright about what is being claimed, and presented in such an easy-to-digest format. Thanks so much for doing this.
Thanks for the kind words!
Re: Measuring the impact of AI on experienced open-source developer productivity
#124Earlier quoted context omitted.
Our largest funding was through The Audacious Project -- you can see an announcement here: https://metr.org/blog/2024-10-09-new-support-through-the-aud... Per our website, “To date, April 2025, we have not accepted compensation from AI companies for the evaluations we have conducted.” You can check out the footnote on this page: https://metr.org/donate
This is really disingenuous when you also say that OpenAI and Anthropic have provided you with access and compute credits (on https://metr.org/about ). Not all payment is cash. Compute credits is still by all means compensation.
Is it "really" disingenuous, or is it just a misinterpretation of what it means to be "compensated for"? Seems more like quibbling to me.
Re: Measuring the impact of AI on experienced open-source developer productivity
#125Earlier quoted context omitted.
They paid the developers about $75k in total to do this so I wouldn't hold your breath!
Neat, how to sign up??
I see these things posted on linkedin. Usually asking $40/hr though. But essentially the same thing as the OP outlines: you do some domain related task assigned either with or without an AI tool. Check linked in. They will have really vague titles like "data scientist" though even though that's not what is being described, its study subject. Maybe set 40/hr as a filter on linkedin and see if you can get a few to come up.
Re: Measuring the impact of AI on experienced open-source developer productivity
#126This study neglects to incorporate the fact that I have forgotten how to write code.
I'm curious what space people are working in where AI does their job entirely. I can use it for parts of code, algorithms, error solving, and maybe sometimes a 'first draft'. But there is no way I could finish an entire piece of software with AI only.
Not a lot of people are empowered to create an entire piece of software. Most are probably in the trenches squashing tickets.
Re: Measuring the impact of AI on experienced open-source developer productivity
#127Early 2025. I imagine the results would be quite different with mid 2025 models and tools.
Re: Measuring the impact of AI on experienced open-source developer productivity
#128I’m not surprised that AI doesn’t help people with 5+ years experience in open source contribution, but I’d imagine most people aren’t claiming AI tools are at senior engineer level yet. Soon once the tools and how people use them improve AI won’t be a hinderance for advanced tasks like this, and soon after AI will be able to do these prs on their own. It’s inevitable given the rate of improvement even since this stu…
Even for senior levels the claim has been that AI will speed up their coding (take it over) so they can focus on higher level decisions and abstract level concepts. These contributions are not those and based on prior predictions the productivity should have gone up.
Re: Measuring the impact of AI on experienced open-source developer productivity
#129N = 16 developers. Is this enough to draw any meaningful conclusions?
That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection.
If you are looking for a 0.1% increase in productivity, then 16 is too small.
Re: Measuring the impact of AI on experienced open-source developer productivity
#130Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. I totally agree with this. Although also, you can end up in a bad spot even after you've gotten pretty good at getting the AI tools to give you good output, because you fail to learn the code you're producing well. A developer gets better at the code they're…
So a really difficult skill in my mind is continually avoiding temptation to vibe.
I agree. I have found that I can use agents most effectively by letting it write code in small steps. After each step I do review of the changes and polish it up (either by doing the fixups myself or prompting). I have found that this helps me understanding the code, but also avoids that the model gets in a bad solution space or produces unmaintainable code.
I also think this kind of close-loop is necessary. Like yesterday I let an LLM write a relatively complex data structure. It got the implementation nearly correct, but was stuck, unable to find an off-by-one comparison. In this case it was easy to catch because I let it write property-based tests (which I had to fix up to work properly), but it's easy for things to slip through the cracks if you don't review carefully.
(This is all using Cursor + Claude 4.)