Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

171–180 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#171

Post by simonw »

blibble wrote: post #169
steveklabnik wrote: post #151

Earlier quoted context omitted.

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.

Re: Measuring the impact of AI on experienced open-source developer productivity

#173

Post by biophysboy »

sarchertech wrote: post #129
AvAn12 wrote: post #118

N = 16 developers. Is this enough to draw any meaningful conclusions?

That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection. If you are looking for a 0.1% increase in productivity, then 16 is too small.

Well it depends on the variance of the random variable itself. You're right that with big, obvious effects, a larger n is less "necessary". I could see individuals having very different "productivities", especially when the idea is flattened down to completion time.

Re: Measuring the impact of AI on experienced open-source developer productivity

#174

Post by WD-42 »

furyofantares wrote: post #72
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. I totally agree with this. Although also, you can end up in a bad spot even after you've gotten pretty good at getting the AI tools to give you good output, because you fail to learn the code you're producing well. A developer gets better at the code they're…

I feel the same way. I use it for super small chunks, still understand everything it outputs, and often manually copy/paste or straight up write myself. I don't know if I'm actually faster before, but it feels more comfy than alt-tabbing to stack overflow, which is what I feel like it's mostly replaced.

Poor stack overflow, it looks like they are the ones really hurting from all this.

Re: Measuring the impact of AI on experienced open-source developer productivity

#175

Post by tcdent »

asdff wrote: post #126
resource_waste wrote: post #62

Earlier quoted context omitted.

I'm curious what space people are working in where AI does their job entirely. I can use it for parts of code, algorithms, error solving, and maybe sometimes a 'first draft'. But there is no way I could finish an entire piece of software with AI only.

Not a lot of people are empowered to create an entire piece of software. Most are probably in the trenches squashing tickets.

I do create entire pieces of software, and while my workflow is always evolving, it goes something like this:

Define schemas, interfaces, and perhaps some base classes that define the attributes I'm thinking about.

Research libraries that support my cause, and include them.

Reference patterns I have established in other parts of the codebase; internal tooling for database, HTTP services, etc.

Instruct the agent to come up with a plan for a first pass at execution in markdown format. Iterate on this plan; "what about X?"

Splat a bunch of code down that supports the structure I'm looking for. Iterate. Cleanup. Iterate. Implement unit tests and get them to pass.

Go back through everything manually and adjust it to suit my personal style, while at the same time fully understanding what's being done and why.

I use STT a lot to have conversations with the agent as we go, and very rarely allow it to make sequential edits without reviewing first; this is a great opportunity to go back and forth and refine what's being written.

Re: Measuring the impact of AI on experienced open-source developer productivity

#176

Post by Terr_ »

Dilettante_ wrote: post #167
sandinmyjoints wrote: post #162

Earlier quoted context omitted.

That’s an interesting adage. Any ideas of its source?

It might have been in Kahneman's "Thinking, Fast and Slow"

I'm not sure, but something involving Kahneman et al. seems very plausible: The relevant term is probably "Attribute Substitution."

https://en.wikipedia.org/wiki/Attribute_substitution

Re: Measuring the impact of AI on experienced open-source developer productivity

#177

Post by asdff »

tcdent wrote: post #175
asdff wrote: post #126

Earlier quoted context omitted.

Not a lot of people are empowered to create an entire piece of software. Most are probably in the trenches squashing tickets.

I do create entire pieces of software, and while my workflow is always evolving, it goes something like this: Define schemas, interfaces, and perhaps some base classes that define the attributes I'm thinking about. Research libraries that support my cause, and include them. Reference patterns I have established in other parts of the codebase; internal tooling for database, HTTP services, etc. Instruct the agent to co…

You are going well above and beyond what a lot of people do to be fair. There are people in senior roles who are just futzing with json files.

Re: Measuring the impact of AI on experienced open-source developer productivity

#178

Post by furyofantares »

bluefirebrand wrote: post #133
furyofantares wrote: post #72

Earlier quoted context omitted.

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. I totally agree with this. Although also, you can end up in a bad spot even after you've gotten pretty good at getting the AI tools to give you good output, because you fail to learn the code you're producing well. A developer gets better at the code they're…

> Take a whole week to do a month's worth of features Everything else in your post is so reasonable and then you still somehow ended up suggesting that LLMs should be quadrupling our output

I'm specifically talking about greenfield work. I do a lot of game prototypes, it definitely does that at the very beginning.

Re: Measuring the impact of AI on experienced open-source developer productivity

#179

Post by akavi »

inetknght wrote: post #42

> We pay developers $150/hr as compensation for their participation in the study. Can someone point me to these 300k/yr jobs?

L5 ("Senior") at any FAANG co, L6 ("Staff") at pretty much any VC-backed startup in the bay.

Re: Measuring the impact of AI on experienced open-source developer productivity

#180

Post by cwillu »

simonw wrote: post #171
blibble wrote: post #169

Earlier quoted context omitted.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.

The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed.

Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true: an academic study does not get funded, executed, written up, and published in less time.

Post reply on HN