Measuring the impact of AI on experienced open-source developer productivity • Hacker News Board

Re: Measuring the impact of AI on experienced open-source developer productivity

#61

Post by lukan » Thu, Jul 10, 2025, 5:53 PM UTC

skydhash wrote: post #48

emodendroket wrote: post #18

Earlier quoted context omitted.

I think it’s most useful when you basically need Stack Overflow on steroids: I basically know what I want to do but I’m not sure how to achieve it using this environment. It can also be helpful for debugging and rubber ducking generally.

The issue is that it is slow and verbose, at least in its default configuration. The amount of reading is non trivial. There’s a reason most references are dense.

Those issues you can partly solve by changing the prompt to tell it to be concise and don't explain its code.

But nothing will make them stick to the one API version I use.

Re: Measuring the impact of AI on experienced open-source developer productivity

#62

Post by resource_waste » Thu, Jul 10, 2025, 5:53 PM UTC

tcdent wrote: post #16

This study neglects to incorporate the fact that I have forgotten how to write code.

I'm curious what space people are working in where AI does their job entirely.

I can use it for parts of code, algorithms, error solving, and maybe sometimes a 'first draft'.

But there is no way I could finish an entire piece of software with AI only.

Re: Measuring the impact of AI on experienced open-source developer productivity

#63

Post by gedy » Thu, Jul 10, 2025, 5:55 PM UTC

mjr00 wrote: post #52

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

> My intuition here is that this study mainly demonstrated that the learning curve on AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learing curve. Definitely. Effective LLM usage is not as straightforward as people believe. Two big things I see a lot of developers do when they share chats: 1. Talk to the LLM like…

> Talk to the LLM like a human

Maybe the LLM doesn't strictly need it, but typing out does bring some clarity for the asker. I've found it helps a lot to catch myself - what am I even wanting from this?

Re: Measuring the impact of AI on experienced open-source developer productivity

#64

Post by simonw » Thu, Jul 10, 2025, 5:55 PM UTC

narush wrote: post #60

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

Hey Simon -- thanks for the detailed read of the paper - I'm a big fan of your OS projects! Noting a few important points here: 1. Some prior studies that find speedup do so with developers that have similar (or less!) experience with the tools they use. In other words, the "steep learning curve" theory doesn't differentially explain our results vs. other results. 2. Prior to the study, 90+% of developers had reasona…

Thanks for the detailed reply! I need to spend a bunch more time with this I think - above was initial hunches from skimming the paper.

Re: Measuring the impact of AI on experienced open-source developer productivity

#65

Post by smokel » Thu, Jul 10, 2025, 5:58 PM UTC

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I notice that some people have become more productive thanks to AI tools, while others are not.

My working hypothesis is that people who are fast at scanning lots of text (or code for that matter) have a serious advantage. Being able to dismiss unhelpful suggestions quickly and then iterating to get to helpful assistance is key.

Being fast at scanning code correlates with seniority, but there are also senior developers who can write at a solid pace, but prefer to take their time to read and understand code thoroughly. I wouldn't assume that this kind of developer gains little profit from typical AI coding assistance. There are also juniors who can quickly read text, and possibly these have an advantage.

A similar effect has been around with being able to quickly "Google" something. I wouldn't be surprised if this is the same trait at work.

Re: Measuring the impact of AI on experienced open-source developer productivity

#66

Post by frotaur » Thu, Jul 10, 2025, 5:58 PM UTC

mjr00 wrote: post #52

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

> My intuition here is that this study mainly demonstrated that the learning curve on AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learing curve. Definitely. Effective LLM usage is not as straightforward as people believe. Two big things I see a lot of developers do when they share chats: 1. Talk to the LLM like…

I'm not sure about your example about talking to LLMs. There is good reason to think that speaking to it like a human might produce better results, as that's what most of the training data is composed of.

I don't have any studies, but it eems to me reasonable to assume.

(Unlike google, where presumably it actually used keywords anyway)

Re: Measuring the impact of AI on experienced open-source developer productivity

#67

Post by narush » Thu, Jul 10, 2025, 5:59 PM UTC

Jabrov wrote: post #2

Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.

Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues (7.5 with AI and 7.5 without AI) on average.

Re: Measuring the impact of AI on experienced open-source developer productivity

#68

Post by uludag » Thu, Jul 10, 2025, 5:59 PM UTC

nestorD wrote: post #45

One thing I could not find on a cursory read is how used were those developers to AI tools. I would expect someone using those regularly to benefit while someone who only played with them a couple of time would likely be slowed down as they deal with the friction of learning to be productive with the tool.

In this case though you still wouldn't necessarily know if the AI tools had a positive causal effect. For example, I practically live in Emacs. Take that away and no doubt I would be immensely less effective. That Emacs improves my productivity and without it I am much worse in no way implies that Emacs is better than the alternatives.

I feel like a proper study for this would involve following multiple developers over time, tracking how their contribution patterns and social standing changes. For example, take three cohorts of relatively new developers: instruct one to go all in on agentic development, one to freely use AI tools, and one prohibited from AI tools. Then teach these developers open source (like a course off of this book: https://pragprog.com/titles/a-vbopens/forge-your-future-with...) and have them work for a year to become part of a project of their choosing. Then in the end, track a number of metrics such as leadership position in community, coding/non-coding contributions, emotional connection to project, social connections made with community, knowledge of code base, etc.

Personally, my prior probability is that the no-ai group would likely still be ahead overall.

Re: Measuring the impact of AI on experienced open-source developer productivity

#69

Post by narush » Thu, Jul 10, 2025, 6:00 PM UTC

simonw wrote: post #64

narush wrote: post #60

Earlier quoted context omitted.

Hey Simon -- thanks for the detailed read of the paper - I'm a big fan of your OS projects! Noting a few important points here: 1. Some prior studies that find speedup do so with developers that have similar (or less!) experience with the tools they use. In other words, the "steep learning curve" theory doesn't differentially explain our results vs. other results. 2. Prior to the study, 90+% of developers had reasona…

Thanks for the detailed reply! I need to spend a bunch more time with this I think - above was initial hunches from skimming the paper.

Sounds great. Looking forward to hearing more detailed thoughts -- my emails in the paper :)

Re: Measuring the impact of AI on experienced open-source developer productivity

#70

Post by onlyrealcuzzo » Thu, Jul 10, 2025, 6:00 PM UTC

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

How were "experienced engineers" defined?

I've found AI to be quite helpful in pointing me in the right direction when navigating an entirely new code-base.

When it's code I already know like the back of my hand, it's not super helpful, other than maybe doing a few automated tasks like refactoring, where there have already been some good tools for a while.