Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

401–410 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#401

Post by KaiserPro »

viraptor wrote: post #313
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

> How many days/weeks you need to use debuggers effectively

I understand your point, but would counter with: gdb isn't marketed as a cuddly tool that can let anyone do anything.

Re: Measuring the impact of AI on experienced open-source developer productivity

#402

Post by TeMPOraL »

carschno wrote: post #394
TeMPOraL wrote: post #391

Earlier quoted context omitted.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users . So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

It's called grassroots marketing. It works particularly well in the context of GenAI because it is fed with esoteric and ideological fragments that overlap with common beliefs and political trends. https://en.wikipedia.org/wiki/TESCREAL Therefore, classical marketing is less dominant, although more present at down-stream sellers.

Right. Let's take a bunch of semi-related groups I don't like, and make up an acronym for them so any of my criticism can be applied to some subset of those groups in some form, thus making it seem legitimate and not just a bunch of half-assed strawman arguments.

Also, I guess you're saying I'm a paid shill, or have otherwise been brainwashed by marketing of the vendors, and therefore my positive experiences with LLMs are a lie? :).

I mean, you probably didn't mean that, but part of my point is that you see those positive reports here on HN too, from real people who've been in this community for a while and are not anonymous Internet users - you can't just dismiss that as "grassroot marketing".

Re: Measuring the impact of AI on experienced open-source developer productivity

#403

Post by lukan »

malfist wrote: post #292
lukan wrote: post #61

Earlier quoted context omitted.

Those issues you can partly solve by changing the prompt to tell it to be concise and don't explain its code. But nothing will make them stick to the one API version I use.

The less verbosity you allow the dumber the LLM is. It thinks in tokens and if you keep it from using tokens it's lobotomized.

It can think as much as it wants and still return just code in the end.

Re: Measuring the impact of AI on experienced open-source developer productivity

#404

Post by sfn42 »

ruszki wrote: post #260
thesz wrote: post #251

Earlier quoted context omitted.

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This is what I heard about strong type systems (especially Haskell's) about 20-15 years ago. "History does not repeat, but it rhymes." If we rhyme "strong types will change the world" with "agentic LLMs will change the world," what do we get? My personal the…

Maybe it depends on the task. I’m 100% sure, that if you think that type system is a drawback, then you have never code in a diverse, large codebase. Our 1.5 million LOC 30 years old monolith would be completely unmaintainable without it. But seriously, anything without a formal type system above 10 LOC after a few years is unmaintainable. An informal is fine for a while, but not long for sure. On a 30 years old code…

It's generally trivial for conventional class-based type systems like those in Java and C#, but TypeScript is a different beast entirely. On the surface it seems similar but it's so much deeper than the others.

I don't like it. I know it is the way it is because it's supposed to support all the cursed weird stuff you can do in JS, but to me as a fullstack developer who's never really taken the time to deep dive and learn TS properly it often feels more like an obstacle. For my own code it's fine, but when I have to work with third party libraries it can be really confusing. It's definitely a skill issue though.

Re: Measuring the impact of AI on experienced open-source developer productivity

#405

Post by intended »

TeMPOraL wrote: post #391
intended wrote: post #365

Earlier quoted context omitted.

> LLMs, especially at the scale we see today The OP qualifies how the marketing cycle for this product is beyond extreme, and its own category. Normal people are being told to worry about AI ending the world, or all jobs disappearing. Simply saying “the problem is the user”, without acknowledging the degree of hype, and expectation setting, the is irresponsible.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users . So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

It is extreme, and on the vendor side. The OpenAI non profit vs profit saga, was about profit seeking vs the future of humanity. People are talking about programming 3.0.

I can appreciate that it’s other users who are saying it’s wrong, but that doesn’t escape the point on ignoring the context.

Moreover, it’s unhelpful communication. Its gives up acknowledging a mutually shared context, the natural confusion that would arise from the ambiguous, high level hype, and the actual down to earth reality.

Even if you have found a way to make it work, having someone understand your workflow can’t happen without connecting the dots between their frame of reference and yours.

Re: Measuring the impact of AI on experienced open-source developer productivity

#406

Post by OccamsMirror »

TeMPOraL wrote: post #391
intended wrote: post #365

Earlier quoted context omitted.

> LLMs, especially at the scale we see today The OP qualifies how the marketing cycle for this product is beyond extreme, and its own category. Normal people are being told to worry about AI ending the world, or all jobs disappearing. Simply saying “the problem is the user”, without acknowledging the degree of hype, and expectation setting, the is irresponsible.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users . So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

I think the relentless podcast blitz by OpenAI and Anthropic founders suggests otherwise. They're both keen to confirm that yes, in 5 - 10 years, no one will have any jobs any more. They're literally out there discussing a post employment world like it's an inevitability.

That's pretty extreme.

Re: Measuring the impact of AI on experienced open-source developer productivity

#407

Post by Avshalom »

sanderjd wrote: post #329
Avshalom wrote: post #323

Earlier quoted context omitted.

Linus did not show up in front of congress talking about how dangerously powerful unregulated version control was to the entirety of human civilization a year before he debuted Git and charged thousands a year to use it.

This seems like a non sequitur. What does this have to do with this thread?

It is completely reasonable to hold cursor/claude to a different standard than gdb or git.

Re: Measuring the impact of AI on experienced open-source developer productivity

#408

Post by lmeyerov »

ivanovm wrote: post #309
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

It's a specialist tool. You wouldn't be surprised that it took awhile for someone to take a big to get at typed programming, parallel programming, docker, IaaC, etc. either.

We have 2 sibling teams, one the genAI devs and the other the regular GPU product devs. It is entirely unsurprising to me that the genAI developers are successfully using coding agents with long-running plans, while the GPU developers are still more at the level of chat-style back-and-forth.

At the same time, everyone sees the potential, and just like other automation movements, are investing in themselves and the code base.

Re: Measuring the impact of AI on experienced open-source developer productivity

#409

Post by lupusreal »

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

A friend of mine, complete non-programmer, has been trying to use ChatGPT to write a phone app. I've been as hands off as I feel I can be, watching how the process goes for him. My observations so far is that it's not going well, he doesn't understand what questions he should be asking so the answers he's getting aren't useful. I encourage him to ask it to teach him the relevant programming but he asks it to help him make the app without programming at all.

With more coaching from me, which I might end up doing, I think he would get further. But I expected the chatbot to get him further through the process than this. My conclusion so far is that this technology won't meaningfully shift the balance of programmers to non-programmers in the general population.

Re: Measuring the impact of AI on experienced open-source developer productivity

#410

Post by fingerlocks »

TeMPOraL wrote: post #392
fingerlocks wrote: post #377

Earlier quoted context omitted.

No one is claiming 10x perf gains in vim. It’s just a fun geeky thing to use with a lot of zany customizations. And after two hellish years of memory muscling enough keyboard bindings to finally be productive, you earned it! It’s a badge of pride! But we all know you’re still fat fingering ggdG on occasion and silently cursing to yourself.

> No one is claiming 10x perf gains in vim. Sure they are - or at least were, unitl the last couple years. Same thing with Emacs. It's hard to claim this now, because the entire industry shifted towards webshit and cloud-based practices across the board, and the classical editors just can't keep up with VS Code. Despite the latter introducing LSP, which leveled the playing field wrt. code intelligence itself, the sur…

Yeah I’m in my 40s and have been using vim for decades. Sure there was an occasional rando stirring up the forums about made-up productivity gains to get some traffic to their blog, but that was it. There has always been push back from many of the strongest vim advocates that the appeal is not about typing speed or whatever it was they were claiming. It’s just ergonomics and power.

It’s just not comparable to the LLM crazy hype train.

And to belabor your other point, I have treesitter, lsp, and GitHub Copilot agent all working flawlessly in neovim. Ts and lsp are neovim builtins now. And it’s custom built for exactly how I want it to be, and none of that blinking shit or nagging dialog boxes all over VSCode.

I have VScode and vim open to the same files all day quite literally side by side, because I work at Microsoft, share my screen often, and there are still people that have violent allergic reactions to a terminal and vim. Vim can do everything VSCode does and it’s not dogshit slow.

Post reply on HN