Measuring the impact of AI on experienced open-source developer productivity • Hacker News Board

Re: Measuring the impact of AI on experienced open-source developer productivity

#361

Post by benreesman » Fri, Jul 11, 2025, 3:44 AM UTC

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I don't even think we know how to do it yet. I revise my whole attitude and all of my beliefs about this stuff every week: I figure out things that seemed really promising don't pan out, I find stuff that I kick myself for not realizing sooner, and it's still this high-stakes game. I still blow a couple of days and wish I had just done it the old-fashioned way, and then I'll catch a run where it's like, fuck, I was never that good, that's the last 5-10% that breaks a PB.

I very much think that these things are going to wind up being massive amplifiers for people who were already extremely sophisticated and then put massive effort into optimizing them and combining them with other advanced techniques (formal methods, top-to-bottom performance orientation).

I don't think this stuff is going to democratize software engineering at all, I think it's going to take the difficulty level so high that it's like back when Djikstra or Tony Hoare was a fairly typical computer programmer.

Re: Measuring the impact of AI on experienced open-source developer productivity

#362

Post by rester324 » Fri, Jul 11, 2025, 3:47 AM UTC

eightysixfour wrote: post #342

rester324 wrote: post #311

Earlier quoted context omitted.

> Interestingly, the best AI assisted devs have often moved to management/solution architecture, and they find the AI code tools brought back some of the love of coding This suggests me though that they are bad at coding, otherwise they would have stayed longer. And I can't find anything in your comment that would corroborate the opposite. So what gives? I am not saying what you say is untrue, but you didn't give any…

> This suggests me though that they are bad at coding, otherwise they would have stayed longer. Or they care about producing value, not just the code, and realized they had more leverage and impact in other roles. > And I can't find anything in your comment that would corroborate the opposite. I didn’t try and corroborate the opposite. Honestly, I don’t care about the “best coders.” I care about people who do their j…

> Honestly, I don’t care about the “best coders.”

> Interestingly, the best AI assisted devs have often moved to management/solution architecture

Is it just me? Or does it seem to others as well that you pretty much rank these people even at the moment and your first comment contradicts your second comment? Especially when you admit that you rank them based on velocity.

I am not saying you shouldn't do that, but it feels to me like rating road construction workers on the number of potholes fixed, even though it's very possible that the potholes are caused by the sloppy work to begin with.

Not what I would want to do.

Re: Measuring the impact of AI on experienced open-source developer productivity

#363

Post by Maxious » Fri, Jul 11, 2025, 3:58 AM UTC

ivanovm wrote: post #309

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

>It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user.

Apple's Response to iPhone 4 Antenna Problem: You're Holding It Wrong https://www.wired.com/2010/06/iphone-4-holding-it-wrong/

Re: Measuring the impact of AI on experienced open-source developer productivity

#364

Post by jeswin » Fri, Jul 11, 2025, 4:12 AM UTC

ivanovm wrote: post #309

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

Not every tool can be figured out in a day (or a week or more). That doesn't mean that the tool is useless, or that the user is incapable.

Re: Measuring the impact of AI on experienced open-source developer productivity

#365

Post by intended » Fri, Jul 11, 2025, 4:21 AM UTC

viraptor wrote: post #313

ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

> LLMs, especially at the scale we see today

The OP qualifies how the marketing cycle for this product is beyond extreme, and its own category.

Normal people are being told to worry about AI ending the world, or all jobs disappearing.

Simply saying “the problem is the user”, without acknowledging the degree of hype, and expectation setting, the is irresponsible.

Re: Measuring the impact of AI on experienced open-source developer productivity

#366

Post by blub » Fri, Jul 11, 2025, 4:33 AM UTC

viraptor wrote: post #313

ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

It is completely typical, but at the same time abnormal to have tools with such poor usability.

A good debugger is very easy to use. I remember the Visual Studio debugger or the C++ debugger on Windows were a piece of cake 20 years ago, while gdb is still painful today. Java and .NET had excellent integrated debuggers while golang had a crap debugging story for so long that I don’t even use a debugger with it. In fact I almost never use debuggers any more.

Version control - same story. CVS for all its problems I had learned to use almost immediately and it had a GUI that was straightforward. git I still have to look up commands for in some cases. Literally all the good git UIs cost a non-trivial amount of money.

Programming languages are notoriously full of unnecessary complexity. Personal pet peeve: Rust lifetime management. If this is what it takes, just use GC (and I am - golang).

Re: Measuring the impact of AI on experienced open-source developer productivity

#367

Post by coffeefirst » Fri, Jul 11, 2025, 4:36 AM UTC

evanelias wrote: post #233

pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

This is fascinating and would go a long ways to explain why people seem to have totally different experiences with the same machines.

Re: Measuring the impact of AI on experienced open-source developer productivity

#368

Post by fiddlerwoaroof » Fri, Jul 11, 2025, 4:51 AM UTC

pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

I think just about every developer hack turns out this way: static vs dynamic types; keyboard shortcuts vs mice; etc. But I think it’s also possible to over-interpret these findings: using the tools that make your work enjoyable has important second-order effects even if they aren’t the productivity silver bullet everyone claims they are.

Re: Measuring the impact of AI on experienced open-source developer productivity

#369

Post by blub » Fri, Jul 11, 2025, 4:56 AM UTC

smokel wrote: post #65

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I notice that some people have become more productive thanks to AI tools, while others are not. My working hypothesis is that people who are fast at scanning lots of text (or code for that matter) have a serious advantage. Being able to dismiss unhelpful suggestions quickly and then iterating to get to helpful assistance is key. Being fast at scanning code correlates with seniority, but there are also senior develope…

One has to take time to review code and think through different aspects of execution (like memory management, concurrency, etc). Plenty of code cannot be scanned.

That said, if the language has GC and other helpers, it makes it easier to scan.

Code and architecture review is an important part of my role and I catch issues that others miss because I spend more time. I did use AI for review (GPT 4.1), but only as an addition, since not reliable enough.

Re: Measuring the impact of AI on experienced open-source developer productivity

#370

Post by narush » Fri, Jul 11, 2025, 5:00 AM UTC

yawnxyz wrote: post #360

narush wrote: post #28

Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093

Does this reproduce for early/mid-career engineers who aren't at the top of their game?

How these results transfer to other settings is an excellent question. Previous literature would suggest speedup -- but I'd be excited to run a very similar methodology in those settings. It's already challenging as models + tools have changed!