Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

101–110 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#101

Post by GoatInGrey »

furyofantares wrote: post #97
Uehreka wrote: post #88

Earlier quoted context omitted.

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. You hit the nail on the head here. I feel like I’ve seen a lot of people trying to make strong arguments that AI coding assistants aren’t useful. As someone who uses and enjoys AI coding assistants, I don’t find this research angle to be… uh… very grounded i…

I think the thing is there IS a learning curve, AND there is a productivity mirage, AND they are immensely useful, AND it is context dependent. All of this leads to a lot of confusion when communicating with people who are having a different experience.

It always comes back to nuance!

Re: Measuring the impact of AI on experienced open-source developer productivity

#102

Post by bgwalter »

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

We have heard variations of that narrative for at least a year now. It is not hard to use these chatbots and no one who was very productive in open source before "AI" has any higher output now.

Most people who subscribe to that narrative have some connection to "AI" money, but there might be some misguided believers as well.

Re: Measuring the impact of AI on experienced open-source developer productivity

#103

Post by 0xmusubi »

I find myself having discussions with AI about different design possibilities and it sometimes comes up with ideas I hadn't thought of or features I wasn't aware of. I wouldn't classify this as "overuse" as I often find the discussions useful, even if it's just to get my thoughts down. This might be more relevant for larger scoped tasks or ones where the programmer isn't as familiar with certain features or libraries though.

Re: Measuring the impact of AI on experienced open-source developer productivity

#104

Post by pferde »

01100011 wrote: post #46
noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

As an old dev this is really all I want: a sort of autocorrect for my syntactical errors to save me a couple compile-edit cycles.

What I want is not autocorrect, because that won't teach me anything. I want it to yell at me loudly and point to the syntactical error.

Autocorrect is a scourge of humanity.

Re: Measuring the impact of AI on experienced open-source developer productivity

#105

Post by [deleted] »

narush wrote: post #67
Jabrov wrote: post #2

Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.

Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues (7.5 with AI and 7.5 without AI) on average.

[deleted]

Re: Measuring the impact of AI on experienced open-source developer productivity

#107

Post by narush »

igorkraw wrote: post #86
narush wrote: post #28

Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093

Could you either release the dataset (raw but anonymized) for independent statistical évaluation or at least add the absolute times of each dev per task to the paper? I'm curious what the absolute times of each dev with/without AI was and whether the one guy with lots of Cursor experience was actually faster than the rest of just a slow typer getting a big boost out of llms Also, cool work, very happy to see actually…

Yep, sorry, meant to post this somewhere but forgot in final-paper-polishing-sprint yesterday!

We'll be releasing anonymized data and some basic analysis code to replicate core results within the next few weeks (probably next, depending).

Our GitHub is here (http://github.com/METR/) -- or you can follow us (https://x.com/metr_evals) and we'll probably tweet about it.

Re: Measuring the impact of AI on experienced open-source developer productivity

#108

Post by lukan »

diggan wrote: post #83
lukan wrote: post #61

Earlier quoted context omitted.

Those issues you can partly solve by changing the prompt to tell it to be concise and don't explain its code. But nothing will make them stick to the one API version I use.

> But nothing will make them stick to the one API version I use. Models trained for tool use can do that. When I use Codex for some Rust stuff for example, it can grep from source files in the directory dependencies are stored, so looking up the current APIs is trivial for them. Same works for JavaScript and a bunch of other languages too, as long as it's accessible somewhere via the tools they have available.

Hm, I never tried codex so far, but quite some other tools and models and none could help me in a consistent way. But I am sceptical, because also if I tell them explicitel, to only use one specific version they might or not might use that, depending on their training corpus and temperature I assume.

Re: Measuring the impact of AI on experienced open-source developer productivity

#109

Post by Uehreka »

TechDebtDevin wrote: post #95
Uehreka wrote: post #88

Earlier quoted context omitted.

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. You hit the nail on the head here. I feel like I’ve seen a lot of people trying to make strong arguments that AI coding assistants aren’t useful. As someone who uses and enjoys AI coding assistants, I don’t find this research angle to be… uh… very grounded i…

Still odd to me that the only vibe coded software that gets aquired are by companies selling tools or want to promote vibe coding.

Pardon my caps, but WHO CARES about acquisitions?!

You’ve been given a dubiously capable genie that can write code without you having to do it! If this thing can build first drafts of those side projects you always think about and never get around to, that in and of itself is useful! If it can do the yak-shaving required to set up those e2e tests you know you should have but never have time for it is useful!

Have it try out all the dumb ideas you have that might be cool but don’t feel worth your time to boilerplate out!

I like to think we’re a bunch of creative people here! Stop thinking about how it can make you money and use it for fun!

Re: Measuring the impact of AI on experienced open-source developer productivity

#110

Post by groos »

One thing I've experienced in trying to use LLMs to code in an existing large code base is that it's _extremely_ hard to accurately describe what you want to do. Oftentimes, you are working on a problem with a web of interactions all over the code and describing the problem to an LLM will take far longer than just doing it manually. This is not the case with generating new (boilerplate) code for projects, which is where users report the most favorable interaction with LLMs.
Post reply on HN