Measuring the impact of AI on experienced open-source developer productivity • Hacker News Board

Re: Measuring the impact of AI on experienced open-source developer productivity

#411

Post by ruszki » Fri, Jul 11, 2025, 8:47 AM UTC

sfn42 wrote: post #404

ruszki wrote: post #260

Earlier quoted context omitted.

Maybe it depends on the task. I’m 100% sure, that if you think that type system is a drawback, then you have never code in a diverse, large codebase. Our 1.5 million LOC 30 years old monolith would be completely unmaintainable without it. But seriously, anything without a formal type system above 10 LOC after a few years is unmaintainable. An informal is fine for a while, but not long for sure. On a 30 years old code…

It's generally trivial for conventional class-based type systems like those in Java and C#, but TypeScript is a different beast entirely. On the surface it seems similar but it's so much deeper than the others. I don't like it. I know it is the way it is because it's supposed to support all the cursed weird stuff you can do in JS, but to me as a fullstack developer who's never really taken the time to deep dive and l…

I agree. Typescript is different for another reason too. They ignore edge cases many times, and because of that you can do really-really nice things with it (when it’s not broken). I wondered a lot of times why Java doesn’t include a few things which would be appropriate even in that world, and the answer is almost always because Java cares about edge cases. There are notes about those in Typescript’s doc or issues.

Re: Measuring the impact of AI on experienced open-source developer productivity

#412

Post by 30minAdayHN » Fri, Jul 11, 2025, 9:01 AM UTC

ares623 wrote: post #395

30minAdayHN wrote: post #21

This study focused on experienced OSS maintainers. Here is my personal experience, but a very different persona (or opposite to the one in the study). I always wanted to contribute to OSS but never had time to. Finally was able to do that, thanks to AI. Last month, I was able to contribute to 4 different repositories which I would never have dreamed of doing it. I was using an async coding agent I built[1], to genera…

Did you make the contributions though? Or did the LLM? This is not directed at you, but I am worried that contributors that use AI "exclusively" to contribute to OSS projects are extracting the value (street cred, being seen as part of the project community) without actually contributing anything (by being one more person that knows the codebase and can help steward it). It's the same thing we've seen out of enshitti…

Did not take it personal. You brought up a good point.

I've slightly alternate perspective. Imo, using OSS without contributing is the value extraction without giving back.

If someone can fix a bunch of chores (that still take human time), with the use of AI (even though they don't become stewards), I still see it as giving back. Of course, there is a value chain - contributing with AI without understanding code is the bottom of value creation. Like you mentioned, also being a steward is the top of the value chain. Along the way, if the contributor builds some sorta reputation that would help with their career or other outcomes, so be it.

So in that sense, I don't see it as enshittification. AI might make a pathway to resolve a bunch of things which otherwise wouldn't be resolved. In fact, this was the line of thinking for the tool we built. Instead of people making these mindless PRs, can we build an agent that can take care of 'trivial' tasks. I manually created PRs to test that hypothesis.

There is also a natural self selection here. If someone was able to fix something without understanding any code, that is also indicative of how trivial the task is. There is a reverse effect to my argument though. These "AI contributors" can create a ton of PRs that would create a lot of work for maintainers to review them.

In my case, I was being upfront about how I'm raising PRs and requesting permissions if it is OK to work on certain issue. Maintainers are quite open and inviting.

Re: Measuring the impact of AI on experienced open-source developer productivity

#413

Post by theshrike79 » Fri, Jul 11, 2025, 9:22 AM UTC

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

LLMs are good for things you know how to do, but can't be arsed to. Like small tools with extensive use of random APIs etc.

For example I whipped together a Steam API -based tool that gets my game library and enriches it with data available in maybe 30 minutes of active work.

The LLM (Cursor with Gemini Pro + Claude 3.7 at the time IIRC) spent maybe 2-3 hours on it while I watched some shows on my main display and it worked on my second screen with me directing it.

Could I have done it myself from scratch like a proper artisan? Most definitely. Would I have bothered? Nope.

Re: Measuring the impact of AI on experienced open-source developer productivity

#414

Post by lompad » Fri, Jul 11, 2025, 9:23 AM UTC

AIorNot wrote: post #256

Hey guys why are we making it so complicated? do we really need a paper and study? anyway -AI as the tech currently stand is a new skill to use and takes us humans time to learn, but once we do well, its becomes force multiplier ie see this: https://claude.ai/public/artifacts/221821f0-0677-409b-8294-3...

Because for now, that's just what those financially profiting from the AI-hype tell us. Be it sama, hyung or nadella, they all profit if people _believe_ AI is a force multiplier for everybody. Reality is much more muddy though and it's absolutely not as obvious as those people claim.

And keep in mind that a 5-10x price hike is to be expected if those companies keep spending billions to make millions.

Right now, there is a consistent stream of papers incoming which indicates that AI might be much more of a specialized tool for very particular situations instead of the "solve everything"-tool the hype makes people believe. This is highly significant.

"Just believe me bro" is just not enough.

Re: Measuring the impact of AI on experienced open-source developer productivity

#415

Post by disgruntledphd2 » Fri, Jul 11, 2025, 9:36 AM UTC

OccamsMirror wrote: post #406

TeMPOraL wrote: post #391

Earlier quoted context omitted.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users . So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

I think the relentless podcast blitz by OpenAI and Anthropic founders suggests otherwise. They're both keen to confirm that yes, in 5 - 10 years, no one will have any jobs any more. They're literally out there discussing a post employment world like it's an inevitability. That's pretty extreme.

Those billions won't raise themselves, you know.

More generally, these execs are talking their book as they're in a low margin capital intensive businesses whose future is entirely dependent on raising a bunch more money, so hype and insane claims are necessary for funding.

Now, maybe they do sortof believe it, but if so, why do they keep hiring software engineers and other staff?

Re: Measuring the impact of AI on experienced open-source developer productivity

#416

Post by carschno » Fri, Jul 11, 2025, 9:39 AM UTC

TeMPOraL wrote: post #402

carschno wrote: post #394

Earlier quoted context omitted.

It's called grassroots marketing. It works particularly well in the context of GenAI because it is fed with esoteric and ideological fragments that overlap with common beliefs and political trends. https://en.wikipedia.org/wiki/TESCREAL Therefore, classical marketing is less dominant, although more present at down-stream sellers.

Right. Let's take a bunch of semi-related groups I don't like, and make up an acronym for them so any of my criticism can be applied to some subset of those groups in some form, thus making it seem legitimate and not just a bunch of half-assed strawman arguments. Also, I guess you're saying I'm a paid shill, or have otherwise been brainwashed by marketing of the vendors, and therefore my positive experiences with LLM…

> I mean, you probably didn't mean that

Correct, I think you've read too much into it. Grassroots marketing is not a pejorative term, either. Its strategy is to trigger positive reviews about your product, ideally by independent, credible community members, indeed.

That implies that those community members have motivations other than being paid. Ideologies and shared beliefs can be some of them. Being happy about the product is a prerequisite, whatever that means for the individual user.

Re: Measuring the impact of AI on experienced open-source developer productivity

#417

Post by ares623 » Fri, Jul 11, 2025, 9:41 AM UTC

30minAdayHN wrote: post #412

ares623 wrote: post #395

Earlier quoted context omitted.

Did you make the contributions though? Or did the LLM? This is not directed at you, but I am worried that contributors that use AI "exclusively" to contribute to OSS projects are extracting the value (street cred, being seen as part of the project community) without actually contributing anything (by being one more person that knows the codebase and can help steward it). It's the same thing we've seen out of enshitti…

Did not take it personal. You brought up a good point. I've slightly alternate perspective. Imo, using OSS without contributing is the value extraction without giving back. If someone can fix a bunch of chores (that still take human time), with the use of AI (even though they don't become stewards), I still see it as giving back. Of course, there is a value chain - contributing with AI without understanding code is t…

Thanks, I like your perspective. Hard to be optimistic these days.

Re: Measuring the impact of AI on experienced open-source developer productivity

#418

Post by Guillaume86 » Fri, Jul 11, 2025, 9:54 AM UTC

narush wrote: post #60

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

Hey Simon -- thanks for the detailed read of the paper - I'm a big fan of your OS projects! Noting a few important points here: 1. Some prior studies that find speedup do so with developers that have similar (or less!) experience with the tools they use. In other words, the "steep learning curve" theory doesn't differentially explain our results vs. other results. 2. Prior to the study, 90+% of developers had reasona…

Using devs working in their own repository is certainly understandable, but it might also explain in part the results. Personally I barely use AI for my own code, while on the other hand when working on some one off script or unfamiliar code base, I get a lot more value from it.

Re: Measuring the impact of AI on experienced open-source developer productivity

#419

Post by staunton » Fri, Jul 11, 2025, 9:59 AM UTC

Avshalom wrote: post #407

sanderjd wrote: post #329

Earlier quoted context omitted.

This seems like a non sequitur. What does this have to do with this thread?

It is completely reasonable to hold cursor/claude to a different standard than gdb or git.

What standard would that be?

Re: Measuring the impact of AI on experienced open-source developer productivity

#420

Post by literalAardvark » Fri, Jul 11, 2025, 10:06 AM UTC

atiedebee wrote: post #218

grey-area wrote: post #98

Earlier quoted context omitted.

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wro…

Let me bring you a third (not necessarily true) interpretation: The developer who has experience using cursor saw a productivity increase not because he became better at using cursor, but because he became worse at not using it.

Became worse is possible

Became worse in 50 hours? Super unlikely