Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

341–350 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#341

Post by phyzome »

lmeyerov wrote: post #203

As someone has been doing hardcore genai for 2+ years, my experience has been, and what we advise internally: * 3 weeks to transition from ai pairing to AI Delegation to ai multitasking. So work gains are mostly week 3+. That's 120+ hours in, as someone pretty senior here. * Speedup is the wrong metric. Think throughput, not latency. Some finite amount of work might take longer, but the volume of work should go up be…

Have you actually measured this?

Because one of the big takeaways from this study is that people are bad at predicting and observing their own time spent.

Re: Measuring the impact of AI on experienced open-source developer productivity

#342

Post by eightysixfour »

rester324 wrote: post #311
eightysixfour wrote: post #270

Earlier quoted context omitted.

I have been teaching people at my company how to use AI code tools, the learning curve is way worse for developers and I have had to come up with some exercises to try and breakthrough the curve. Some seemingly can’t get it. The short version is that devs want to give instructions instead of ask for what outcome they want. When it doesn’t follow the instructions, they double down by being more precise, the worst thin…

> Interestingly, the best AI assisted devs have often moved to management/solution architecture, and they find the AI code tools brought back some of the love of coding This suggests me though that they are bad at coding, otherwise they would have stayed longer. And I can't find anything in your comment that would corroborate the opposite. So what gives? I am not saying what you say is untrue, but you didn't give any…

> This suggests me though that they are bad at coding, otherwise they would have stayed longer.

Or they care about producing value, not just the code, and realized they had more leverage and impact in other roles.

> And I can't find anything in your comment that would corroborate the opposite.

I didn’t try and corroborate the opposite.

Honestly, I don’t care about the “best coders.” I care about people who do their job well, sometimes that is writing amazing code but most of the time it isn’t. I don’t have any devs in my company who work in a magical vacuum where they are handed perfectly written tasks, they complete them, and then they do the next one.

If I did, I could replace them with AI faster.

> Also, you didn't define the criteria of getting better. Getting better in terms of what exactly?

Delivery velocity - bug fixes, features, etc. that pass testing/QA and goes to prod.

Re: Measuring the impact of AI on experienced open-source developer productivity

#343

Post by bluefirebrand »

ummonk wrote: post #241
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

Devil's advocate: it's also possible the one developer hasn't become more productive with Cursor, but rather has atrophied their non-AI productivity due to becoming reliant on Cursor.

I suspect you're onto something here but I also think it would be an extremely dramatic atrophy to have occurred in such a short period of time...

Re: Measuring the impact of AI on experienced open-source developer productivity

#344

Post by bluefirebrand »

Aurornis wrote: post #253
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

> A quarter of the participants saw increased performance, 3/4 saw reduced performance. The study used 246 tasks across 16 developers, for an average of 15 tasks per developer. Divide that further in half because tasks were assigned as AI or not-AI assisted, and the sample size per developer is still relatively small. Someone would have to take the time to review the statistics, but I don’t think this is a case where…

> potential confounding effect that AI-enthusiastic developers could actually lose some of their practice in writing code without the tools

I don't think this is a confounding effect

This is something that we definitely need to measure and be aware of, if there is a risk of it

Re: Measuring the impact of AI on experienced open-source developer productivity

#345

Post by phyzome »

mwigdahl wrote: post #302
OpenSourceWard wrote: post #172

Very cool work! And I love the nuance in your methodology and findings. Anyway, I'm preparing myself for all the "Bombshell news: AI is slowing down developers" posts that are coming.

Plus the gaslighting to follow for anyone claiming AI improved their productivity.

Well, it would be a nice counterweight to all the gaslighting of people who claim AI doesn't improve their productivity...

Re: Measuring the impact of AI on experienced open-source developer productivity

#346

Post by gexla »

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

In addition to the learning curve of the tooling, there's also the learning curve of the models. Each have a certain personality that you have to figure out so that you can catch the failure patterns right away.

Re: Measuring the impact of AI on experienced open-source developer productivity

#348

Post by maxbond »

jstummbillig wrote: post #258
card_zero wrote: post #254

Earlier quoted context omitted.

Sorry, just happened to. Slightly rude of me.

Ah, you do you. It's just a fairly kindergarten thing to point out and not something I was actively trying to hide. Whatever it was. Generally, I do a couple of edits for clarity after posting and reading again. Sometimes that involves removing something that I feel could have been said better. If it does not work, I will just delete the comment. Whatever it was must not have been a super huge deal (to me).

FYI there's a "delay" setting in your profile that allows you to make your comment invisible for up to ten minutes.

Re: Measuring the impact of AI on experienced open-source developer productivity

#349

Post by phyzome »

afro88 wrote: post #127

Early 2025. I imagine the results would be quite different with mid 2025 models and tools.

If they used mid 2025 models and tools, the paper would have come out in late 2025, and you would have had the same complaint.
Post reply on HN