Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

391–400 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#391

Post by TeMPOraL »

intended wrote: post #365
viraptor wrote: post #313

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

> LLMs, especially at the scale we see today The OP qualifies how the marketing cycle for this product is beyond extreme, and its own category. Normal people are being told to worry about AI ending the world, or all jobs disappearing. Simply saying “the problem is the user”, without acknowledging the degree of hype, and expectation setting, the is irresponsible.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users. So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

Re: Measuring the impact of AI on experienced open-source developer productivity

#392

Post by TeMPOraL »

fingerlocks wrote: post #377
sanderjd wrote: post #328

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Maybe, but it isn't hard to think of developer tools where this is the case. This is the entire history of editor and IDE wars. Imagine running this same study design with vim. How well would you expect the not-previously-experienced developers to perform in such a study?

No one is claiming 10x perf gains in vim. It’s just a fun geeky thing to use with a lot of zany customizations. And after two hellish years of memory muscling enough keyboard bindings to finally be productive, you earned it! It’s a badge of pride! But we all know you’re still fat fingering ggdG on occasion and silently cursing to yourself.

> No one is claiming 10x perf gains in vim.

Sure they are - or at least were, unitl the last couple years. Same thing with Emacs.

It's hard to claim this now, because the entire industry shifted towards webshit and cloud-based practices across the board, and the classical editors just can't keep up with VS Code. Despite the latter introducing LSP, which leveled the playing field wrt. code intelligence itself, the surrounding development process and the ecosystem increasingly demands you use web-based or web-derived tools and practices, which all see a browser engine as a basic building block. Classical editors can't match the UX/DX on that, plus the whole thing breaks basic assumptions about UI that were the source of the "10x perf gains" in vim and Emacs.

Ironically, a lot of the perf gains from AI come from letting you avoid dealing with the brokenness of the current tools and processes, that vim and Emacs are not equipped to handle.

Re: Measuring the impact of AI on experienced open-source developer productivity

#393

Post by TeMPOraL »

Maxious wrote: post #363
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

>It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Apple's Response to iPhone 4 Antenna Problem: You're Holding It Wrong https://www.wired.com/2010/06/iphone-4-holding-it-wrong/

The important difference is that in your example, it was the manufacturer telling customers they're holding it wrong. With LLMs, the vendors say no such things - it's the actual users that are saying this to their peers.

Re: Measuring the impact of AI on experienced open-source developer productivity

#394

Post by carschno »

TeMPOraL wrote: post #391
intended wrote: post #365

Earlier quoted context omitted.

> LLMs, especially at the scale we see today The OP qualifies how the marketing cycle for this product is beyond extreme, and its own category. Normal people are being told to worry about AI ending the world, or all jobs disappearing. Simply saying “the problem is the user”, without acknowledging the degree of hype, and expectation setting, the is irresponsible.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users . So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

It's called grassroots marketing. It works particularly well in the context of GenAI because it is fed with esoteric and ideological fragments that overlap with common beliefs and political trends. https://en.wikipedia.org/wiki/TESCREAL

Therefore, classical marketing is less dominant, although more present at down-stream sellers.

Re: Measuring the impact of AI on experienced open-source developer productivity

#395

Post by ares623 »

30minAdayHN wrote: post #21

This study focused on experienced OSS maintainers. Here is my personal experience, but a very different persona (or opposite to the one in the study). I always wanted to contribute to OSS but never had time to. Finally was able to do that, thanks to AI. Last month, I was able to contribute to 4 different repositories which I would never have dreamed of doing it. I was using an async coding agent I built[1], to genera…

Did you make the contributions though? Or did the LLM?

This is not directed at you, but I am worried that contributors that use AI "exclusively" to contribute to OSS projects are extracting the value (street cred, being seen as part of the project community) without actually contributing anything (by being one more person that knows the codebase and can help steward it).

It's the same thing we've seen out of enshittification of everything. Value extraction without giving back.

Maybe I'm too much of a cynic. Maybe majority of OSS projects don't care. But I know I will be saddened if one of the OSS projects I care about get taken over by such "value extractors".

Re: Measuring the impact of AI on experienced open-source developer productivity

#396

Post by thesz »

leshow wrote: post #272
thesz wrote: post #251

Earlier quoted context omitted.

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This is what I heard about strong type systems (especially Haskell's) about 20-15 years ago. "History does not repeat, but it rhymes." If we rhyme "strong types will change the world" with "agentic LLMs will change the world," what do we get? My personal the…

I don't think that's a fair comparison. Type systems don't produce probabilistic output. Their entire purpose is to reduce the scope of possible errors you can write. They kind of did change the world, didn't they? I mean, not everyone is writing Haskell but Rust exists and it's doing pretty well. There was also not really a case to be made where type systems made software in general _worse_. But you could definitely…

That probabilistic output has to be symbolically constrained - SQL/JSON/other code is generated through syntax constrained beam search.

You brought up Rust, it is fascinating.

The Rust's type system differs from typical Hindle-Milner by having operations that can remove definitions from environment of the scope.

Rust was conceived in 2006.

In 2006 there already were HList papers by Oleg Kiselyov [1] that had shown how to keep type level key-value lists with addition, removal and lookup, and type-level stateful operations like in [2] were already possible, albeit, most probably, not with nice monadic syntax support.

  [1] https://okmij.org/ftp/Haskell/HList-ext.pdf
  [2] http://blog.sigfpe.com/2009/02/beyond-monads.html
It was entirely possible to have prototype Rust to be embedded into Haskell and have borrow checker implemented as type-level manipulation over double parameterized state monad.

But it was not, Rust was not embedded into Haskell and now it will never get effects (even as weak as monad transformers) and, as a consequence, will never get proper high performance software transactional memory.

So here we are: everything in Haskell's strong type system world that would make Rust better was there at the very beginning of the Rust journey, but had no impact on Rust.

Rhyme that with LLM.

Re: Measuring the impact of AI on experienced open-source developer productivity

#397

Post by DanielVZ »

ivanovm wrote: post #309
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user.

Sorry to be pedantic but this is really common in tech products: vim, emacs, any second-brain app, effectiveness of IDEs depending on learning its features, git, and more.

Re: Measuring the impact of AI on experienced open-source developer productivity

#398

Post by oytis »

sanderjd wrote: post #328
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Maybe, but it isn't hard to think of developer tools where this is the case. This is the entire history of editor and IDE wars. Imagine running this same study design with vim. How well would you expect the not-previously-experienced developers to perform in such a study?

What I like about IDE wars is that it remained a dispute between engineers. Some engineers like fancy pants IDEs and use them, some are good with vim and stick with that. No one ever assumed that Jetbrains autocomplete is going to replace me or that I am outdated for not using it - even if there might be a productivity cost associated with that choice.

Re: Measuring the impact of AI on experienced open-source developer productivity

#399

Post by thesz »

ruszki wrote: post #260
thesz wrote: post #251

Earlier quoted context omitted.

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This is what I heard about strong type systems (especially Haskell's) about 20-15 years ago. "History does not repeat, but it rhymes." If we rhyme "strong types will change the world" with "agentic LLMs will change the world," what do we get? My personal the…

Maybe it depends on the task. I’m 100% sure, that if you think that type system is a drawback, then you have never code in a diverse, large codebase. Our 1.5 million LOC 30 years old monolith would be completely unmaintainable without it. But seriously, anything without a formal type system above 10 LOC after a few years is unmaintainable. An informal is fine for a while, but not long for sure. On a 30 years old code…

Contrarily I believe that strong type system is a plus. Please, look at my other comment: https://news.ycombinator.com/item?id=44529347

My original point was about history and about how can we extract possible outcome from it.

My other comment tries to amplify that too. Type systems were strong enough for several decades now, had everything Rust needed and more years before Rust began, yet they have little penetration into real world, example being that fancy dandy Rust language.

Re: Measuring the impact of AI on experienced open-source developer productivity

#400

Post by ndsipa_pomu »

DanielVZ wrote: post #397
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Sorry to be pedantic but this is really common in tech products: vim, emacs, any second-brain app, effectiveness of IDEs depending on learning its features, git, and more.

Well, surely vim is easy to use - I started it and and haven't stopped using it yet (one day I'll learn how to exit)
Post reply on HN