Measuring the impact of AI on experienced open-source developer productivity • Hacker News Board

Re: Measuring the impact of AI on experienced open-source developer productivity

#461

Post by jpc0 » Fri, Jul 11, 2025, 9:13 PM UTC

mattmanser wrote: post #276

steveklabnik wrote: post #215

Earlier quoted context omitted.

Yes. In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan. As with anything, your miles may vary: I’m not here to tell anyone that thinks they still suck that their experience is invalid, but to me it’s been a pretty big swing.

Ok, I'll have to try it out then. I've got a side project I've 3/4 finished and will let it loose on it. So are you using Claude Code via the max plan, Cursor, or what? I think I'd definitely hit AI news exhaustion and was viewing people raving about this agentic stuff as yet more AI fanbois. I'd just continued using the AI separate as setting up a new IDE seemed like too much work for the fractional gains I'd been s…

Takes this with a massive grain of salt but my experience with Google Code CLI recently, we pay for google products but not others internally, I can’t change that decision.

I asked it two implement two bicubic filters, a high pass filter and a high shelf filter. Some context, using the gemini webapp it would split out the exact code I need with the interfaces I require one shot because this is truly trivial C++ code to write.

15 million tokens and an hour and a half later I now had a project that could not build, the filters were not implemented and my trust in AI agentic workflows broken.

It cost me nothing, I just reset the repo and I was watching youtube videos for that hour and a half.

Your mileage may vary and I’m very sure if this was golang or typescript it might have done significantly better, but even compared to the exact same model in a chat interface my experience was horrible.

I’m sticking to the slightly “worse” experience of using the chat interface which does give me significant improvements in productivity vs letting the agent burn money and time and not produce working code.

Re: Measuring the impact of AI on experienced open-source developer productivity

#462

Post by jpc0 » Fri, Jul 11, 2025, 9:39 PM UTC

partdavid wrote: post #282

mrwaffle wrote: post #170

My overall concern has to do with our developer ecosystem from the important points mentioned by simonw and narush. I've been concerned about this for years but AI reliance seems to be pouring jet fuel on the fire. Particularly troubling is the lack of understanding less-experienced devs will have over time. Does anyone have a counter-argument for this they can share on why this is a good thing?

The shallow analogy is like "why worry about not being able to do arithmetic without a calculator"? Like... the dev of the future just won't need it. I feel like programming has become increasingly specialized and even before AI tool explosion, it's way more possible to be ignorant of an enormous amount of "computing" than it used to be. I feel like a lot of "full stack" developers only understand things to the margi…

> Do we all need to know about processor architectures and microcode and L2 caches and paging and OS distributions and system software…

Have you used modern software… or just software in general to be honest.

We have had orders of magnitude improvement in hardware performance and much fewer orders of magnitude increase in software performance and features.

May I present the windows start menu as a perfect exhibit, we put a web browser in there and made actually finding the software you want to use harder than ever, even search is completely broken 99% of the time (really try powertoys run or even windows + s for a night and day difference).

We add boundless complexity to things that doesn’t need it, millions of lines of code, then waste millions of cycles running security tools to heuristically prevent malicious actors from exploiting our millions of lines of code that is impossible to know because it is deemed to difficult to learn the underlying semantics of the problem domain.

Re: Measuring the impact of AI on experienced open-source developer productivity

#463

Post by iLemming » Fri, Jul 11, 2025, 9:53 PM UTC

fingerlocks wrote: post #457

iLemming wrote: post #452

Earlier quoted context omitted.

> vim and Emacs are not equipped to handle. You clearly don't have a slightest idea of what you're talking about. Emacs is actually still amazing in the LLM era. Language is all about plain text. Plain text remains crucial and will remain important because it's human-readable, machine-parsable, version-control friendly, lightweight and fast, platform-independent, and resistant to obsolescence. Even when analyzing hug…

At first I thought you were replying to me and this was a revival of the old vim + emacs wars. I’m so glad we’re past that now and can join forces against a common enemy. Thank you brother.

There weren't any true "wars" to begin with. The entire thing is just absurd. These ideas are not even in competition, it's like arguing whether a piano or sheet music is "better".

Emacs veterans simply rejected the entire concept of modality, without even trying to understand what it is about. Emacs is inherently a modal editor. Key-chords are stateful, Transient menus (i.e. Magit) are modals, completion is a modal, isearch, dired, calc, C-u (universal argument), recursive editing — these are all modals. What the idea of vim-motions offers is a universal, simplified, structured language to deal with modality, that's all.

Vim users on the other hand keep saying "there's no such thing as vim-mode". And to a certain degree they are right — no vim plugin outside of vim/neovim implements all the features — IdeaVim, VSCode vim plugins, Sublime, etc. - all of them are full of holes and glaring deficiencies. With one notable exception — Evil-mode in Emacs. It is so wonderfully implemented, you wouldn't even notice that it is a plugin, an afterthought. It really does feel like a baked-in, native feature of the editor.

There are no "wars" in our industry — pretty much only misunderstanding, misinterpretation and misuse of certain ideas. It's not even technological — who knows, maybe it's not even sociotechnological. People simply like talking past each other, defending different values without acknowledging they're optimizing for different things.

It's not Vim's, Emacs' or VSCode's fault that we suffer from identity investment - we spend hundreds of hours using one so it becomes our identity. We suffer from simplification impulse — we just love binary choices, we constantly have the nagging "which is better?" question, even when it makes little sense. We're predisposed to tribal belonging — having a common enemy creates in-group cohesion.

But real, experienced craftspeople... they just use whatever works best for them in a given context. That's what we all should strive for — discover old and new ideas, study them, identify good ones, borrow them, shelve the bad ones (who knows, maybe in a different context they may still prove useful). Most importantly, use whatever makes you and your teammates happy. It's far more important than being more productive or being decisively right. If thy stupid thing works, perhaps it ain't that stupid?

Re: Measuring the impact of AI on experienced open-source developer productivity

#464

Post by thesz » Fri, Jul 11, 2025, 10:15 PM UTC

59nadir wrote: post #458

thesz wrote: post #246

What is interesting here is that all predictions were positive, but results are negative. This shows that everyone in the study (economic experts, ML experts and even developers themselves, even after getting experience) are novices if we look at them from the Dunning-Kruger effect [1] perspective. [1] https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect "The Dunning–Kruger effect is a cognitive bias in which…

> "The Dunning–Kruger effect is a cognitive bias in which people with limited competence in a particular domain overestimate their abilities." No, they underestimated their own abilities for the most part; the estimates for AI-disallowed tasks were all undershot in terms of real implementation time. What they overestimated was the ability of LLMs to provide real productivity gains on a given task.

  > What they overestimated was the ability of LLMs to provide real productivity gains on a given task.

This is exactly my point.

This is not about the ability of LLM overestimated by developers, this is about the ability of developer interacting with LLM overestimated by developers themselves, economic experts and ML experts.

LLMs are not "able" per se, they are "prompted" to be "able." They are not agents, but behave as agents on someone's behalf - and no one have a clue whether use of LLMs is positive or detrimental, with the bias being "LLM's use is net positive".

The overestimation of LLM's abilities by everyone calls for Dunning-Kruger.

Re: Measuring the impact of AI on experienced open-source developer productivity

#465

Post by iLemming » Fri, Jul 11, 2025, 10:41 PM UTC

uludag wrote: post #68

nestorD wrote: post #45

One thing I could not find on a cursory read is how used were those developers to AI tools. I would expect someone using those regularly to benefit while someone who only played with them a couple of time would likely be slowed down as they deal with the friction of learning to be productive with the tool.

In this case though you still wouldn't necessarily know if the AI tools had a positive causal effect. For example, I practically live in Emacs. Take that away and no doubt I would be immensely less effective. That Emacs improves my productivity and without it I am much worse in no way implies that Emacs is better than the alternatives. I feel like a proper study for this would involve following multiple developers ov…

FWIW, LLM tooling for Emacs is great. gptel for example allows you to converse with wide-range of different models from anywhere in Emacs — you can spontaneously send requests while typing some text or even browsing M-x menu. I often do things like "summarize current paragraph in pdf document" or "create a few anki cards based on this web page content", etc.

Re: Measuring the impact of AI on experienced open-source developer productivity

#466

Post by lackoftactics » Fri, Jul 11, 2025, 11:02 PM UTC

ivanovm wrote: post #309

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

Stay tuned, a new study is coming with another revelation: you aren't getting faster by using Vim when you are learning it.

My previous employer didn't even allow me to use Vim until I learned it properly so it wouldn't affect my productivity. Why would using a cursor automatically make you better at something if it's just new to you and you are already an elite programmer according to this study?

Re: Measuring the impact of AI on experienced open-source developer productivity

#467

Post by sarchertech » Fri, Jul 11, 2025, 11:47 PM UTC

malfist wrote: post #320

jwrallie wrote: post #303

Earlier quoted context omitted.

You could go the same way as the study, flip a coin to use AI or not, write down the task you just did, the time you thought the task took you and the actual clock time. Repeat and self-evaluate.

Sample size of 16 is already hard enough to draw conclusions from. Sample size of 1 is even worse.

Sample of 16 is plenty if the effect is big enough.

It’s also not a sample size of 1, it’s a sample size of however many tasks you do because you don’t care about measuring the effect AI has on anyone but yourself if you’re trying to discern how it impacts you.

Re: Measuring the impact of AI on experienced open-source developer productivity

#468

Post by patrakov » Fri, Jul 11, 2025, 11:56 PM UTC

TeMPOraL wrote: post #391

intended wrote: post #365

Earlier quoted context omitted.

> LLMs, especially at the scale we see today The OP qualifies how the marketing cycle for this product is beyond extreme, and its own category. Normal people are being told to worry about AI ending the world, or all jobs disappearing. Simply saying “the problem is the user”, without acknowledging the degree of hype, and expectation setting, the is irresponsible.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users . So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

> And it's not the marketing that's saying "you're using it wrong" - it's other users.

No, it's the non-coding managers who vibe-coded a half-working prototype, not other users. And here, the Dunning-Kruger effect is at play - those non-coding types do not understand that AI is not working for them either.

Full disclosure: I do rely on vibe-coded jq lines in one-off scripts that will definitely not process more data after the single intended use, and this is where AI saves my time.

Re: Measuring the impact of AI on experienced open-source developer productivity

#469

Post by patrakov » Sat, Jul 12, 2025, 12:09 AM UTC

OccamsMirror wrote: post #406

TeMPOraL wrote: post #391

Earlier quoted context omitted.

AI marketing isn't extreme - not on the LLM vendor side, at least; the hype is generated downstream of it, for various reasons. And it's not the marketing that's saying "you're using it wrong" - it's other users . So, unless you believe everyone reporting good experience with LLMs is a paid shill, there might actually be some merit to it.

I think the relentless podcast blitz by OpenAI and Anthropic founders suggests otherwise. They're both keen to confirm that yes, in 5 - 10 years, no one will have any jobs any more. They're literally out there discussing a post employment world like it's an inevitability. That's pretty extreme.

This was present (in a positive way, though) even in Soviet films for children.

    Позабыты хлопоты,
    Остановлен бег,
    Вкалывают роботы,
    Счастлив человек!

    Worries forgotten,
    The treadmill doesn't run,
    Robots are working,
    Humans have fun!

Re: Measuring the impact of AI on experienced open-source developer productivity

#470

Post by xandrius » Sat, Jul 12, 2025, 12:32 AM UTC

iLemming wrote: post #460

xandrius wrote: post #432

Earlier quoted context omitted.

On the other hand if you don't use vim, emacs, and other spawns from hell, you get labeled a noob and nothing can ever be said about their terrible UX. I think we can be more open minded that an absolutely brand new technology (literally did not exist 3y ago) might require some amount of learning and adjusting, even for people who see themselves as an Einstein if only they wished to apply themselves.

> you get labeled a noob No one would call one a noob for not using Vim or Emacs. But they might for a different reason. If someone blindly rejects even the notion of these tools without attempting to understand the underlying ideas behind them, that certainly suggests the dilettante nature of the person making the argument. The idea of vim-motions is a beautiful, elegant, pragmatic model. Thinking that it is somehow…

One could try to be poetic with LLMs in order to make their point stronger and still convince absolutely no one who wasn't already convinced.

I'm sure nobody really reject the notion of LLMs but sure as hell do like to moan if the new technology doesn't absolutely perfect fit their own way of working. Does that make them any different than people wanting an editor which is intuitive to use? Nobody will ever know.