Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

281–290 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#281

Post by bix6 »

blibble wrote: post #169
steveklabnik wrote: post #151

Earlier quoted context omitted.

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

Everything actually got better. Look at the image generation improvements as an easily visible benchmark.

I do not program for my day job and I vibe coded two different web projects. One in twenty mins as a test with cloudflare deployment having never used cloudflare and one in a week over vacation (and then fixed a deep safari bug two weeks later by hammering the LLM). These tools massively raise the capabilities for sub-average people like me and decrease the time / brain requirements significantly.

I had to make a little update to reset the KV store on cloudflare and the LLM did it in 20s after failing the syntax twice. I would’ve spent at least a few minutes looking it up otherwise.

Re: Measuring the impact of AI on experienced open-source developer productivity

#282

Post by partdavid »

mrwaffle wrote: post #170

My overall concern has to do with our developer ecosystem from the important points mentioned by simonw and narush. I've been concerned about this for years but AI reliance seems to be pouring jet fuel on the fire. Particularly troubling is the lack of understanding less-experienced devs will have over time. Does anyone have a counter-argument for this they can share on why this is a good thing?

The shallow analogy is like "why worry about not being able to do arithmetic without a calculator"? Like... the dev of the future just won't need it.

I feel like programming has become increasingly specialized and even before AI tool explosion, it's way more possible to be ignorant of an enormous amount of "computing" than it used to be. I feel like a lot of "full stack" developers only understand things to the margin of their frameworks but above and below it they kind of barely know how a computer works or what different wire protocols actually are or what an OS might actually do at a lower level. Let alone the context in which in application sits beyond let's say, a level above a kubernetes pod and a kind of trial-end-error approach to poking at some YAML templates.

Do we all need to know about processor architectures and microcode and L2 caches and paging and OS distributions and system software and installers and openssl engines and how to make sure you have the one that uses native instructions and TCP packets and envoy and controllers and raft systems and topic partitions and cloud IAM and CDN and DNS? Since that's not the case--nearly everyone has vast areas of ignorance yet still does a bunch of stuff--it's harder to sell the idea that whatever AI tools are doing that we lose skills in will somehow vaguely matter in the future.

I kind of miss when you had to know a little of everything and it also seemed like "a little bit" was a bigger slice of what there was to know. Now you talk to people who use a different framework in your own language and you feel like you're talking to deep specialists whose concerns you can barely understand the existence of, let alone have an opinion on.

Re: Measuring the impact of AI on experienced open-source developer productivity

#283

Post by jbeninger »

kennywinker wrote: post #7
dash2 wrote: post #4

The authors say "High developer familiarity with repositories" is a likely reason for the surprising negative result, so I wonder if this generalizes beyond that.

Like if it generalizes to situations where the developer is not familiar with the repo? That doesn’t seem like generalizing, that seems like specifying. Am I wrong in saying that the majority of developer time is spent in repos that they’re familiar with? Every job and project I’ve worked has been on a fixed set of repos the entire time. If AI is only helpful for the first week or two on a project, that’s not very ma…

I'd say I write the majority of my code in areas I'm familiar with, but spend the majority of my _time_ on sections I'm not familiar with, and ai helps a lot more with the latter than the former. I've always felt my coding life is speeding through a hundred lines of easy code then getting stuck on the 101st. Then as I get more experienced that hundred becomes 150, then 200, but always speeding through the easy part until I have to learn something new.

So I never feel like I'm getting any faster. 90% of my time is still spent in frustration, even when I'm producing twice the code at higher quality

Re: Measuring the impact of AI on experienced open-source developer productivity

#284

Post by codyb »

So slow until a learning curve is hit (or as one user posited "until you forget how to work without it").

But isn't the important thing to measure... how long does it take to debug the resulting code at 3AM when you get a PagerDuty alert?

Similarly... how about the quality of this code over time? It's taken a lot of effort to bring some of the code bases I work in into a more portable, less coupled, more concise state through the hard work of

- bringing shared business logic up into shared folders

- working to ensure call chains flow top down towards root then back up through exposed APIs from other modules as opposed to criss-crossing through the directory structure

- working to separate business logic from API logic from display logic

- working to provide encapsulation through the use of wrapper functions creating portability

- using techniques like dependency injection to decouple concepts allowing for easier testing

etc

So, do we end up with better code quality that ends up being more maintainable, extensible, portable, and composable? Or do we just end up with lots of poor quality code that eventually grows to become a tangled mess we spend 50% of our time fighting bugs on?

Re: Measuring the impact of AI on experienced open-source developer productivity

#285

Post by cutemonster »

card_zero wrote: post #231
atiedebee wrote: post #218

Earlier quoted context omitted.

Let me bring you a third (not necessarily true) interpretation: The developer who has experience using cursor saw a productivity increase not because he became better at using cursor, but because he became worse at not using it.

Or, one person in 16 has a particular personality, inclined to LLM dependence.

Didn't they rather mean:

Developers' own skills might atrophy, when they don't write that much code themselves, relying on AI instead.

And now when comparing with/without AI they're faster with. But a year ago they might have been that fast or faster without an AI.

I'm not saying that that's how things are. Just pointing out another way to interpret what GP said

Re: Measuring the impact of AI on experienced open-source developer productivity

#286

Post by jstummbillig »

leshow wrote: post #264
jstummbillig wrote: post #194

Earlier quoted context omitted.

Convenient for whom and what...? There is nothing tangible to gain from you believing or not believing that someone else does (or does not) get a productivity boost from AI. This is not a religion and it's not crypto. The AI users' net worth is not tied to another ones use of or stance on AI (if anything, it's the opposite). More generally, the phenomenon this is quite simply explained and nothing surprising: New thi…

I think you're missing the broader context. There is a lot of people very invested in the maximalist outcome which does create pressure for people to be boosters. You don't need a digital token for that to happen. There's a social media aspect as well that creates a feedback loop about claims. We're in a hype cycle, and it means we should be extra critical when evaluating the tech so we don't get taken in by exaggera…

I mostly don't agree. Yes, there is always social pressure with these things, and we are in a hype cycle, but the people "buying in" are simply not doing much at all. They are mostly consumers, waiting for the next model, which they have no control over or stake in creating (by and large).

The people not buying into the hype, on the other hands, are actually the ones that have a very good reason to be invested, because if they turn out to be wrong they might face some very uncomfortable adjustments in the job landscape and a lot of the skills that they worked so hard to gain and believed to be valuable.

As always, be weary of any claims, but the tension here is very much the reverse of crypto and I don't think that's very appreciated.

Re: Measuring the impact of AI on experienced open-source developer productivity

#287

Post by rs186 »

robwwilliams wrote: post #224
grey-area wrote: post #98

Earlier quoted context omitted.

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wro…

Or a sampling artifact. 4 vs 12 does seem significant within a study, but consider a set of N such studies. I assume that many large companies have tested efficiency gains and losses of there programmers much more extensively than the authors of this tiny study. A survey of companies and their evaluation and conclusions would carry more weight—-excluding companies selling AI products, of course.

If you use binomial test, P(X<=4) is about 0.105 which means p = 0.21.

Re: Measuring the impact of AI on experienced open-source developer productivity

#288

Post by joks »

asdff wrote: post #126
resource_waste wrote: post #62

Earlier quoted context omitted.

I'm curious what space people are working in where AI does their job entirely. I can use it for parts of code, algorithms, error solving, and maybe sometimes a 'first draft'. But there is no way I could finish an entire piece of software with AI only.

Not a lot of people are empowered to create an entire piece of software. Most are probably in the trenches squashing tickets.

I think the question still stands.

Re: Measuring the impact of AI on experienced open-source developer productivity

#289

Post by [deleted] »

burnte wrote: post #214
grey-area wrote: post #98

Earlier quoted context omitted.

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wro…

> Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wrong direction on how useful they are. I would argue you don't need the "as a programming assistant" phrase as right now from my experience over the past 2 years, literally every single AI tool is massively oversold as to its utility. I've literally not seen a single one that d…

[deleted]

Re: Measuring the impact of AI on experienced open-source developer productivity

#290

Post by Aeolun »

blibble wrote: post #169
steveklabnik wrote: post #151

Earlier quoted context omitted.

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

It’s true though? Previous models could do well in specifically created settings. You can throw practically everything at Opus, and it’ll work mostly fine.
Post reply on HN