Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

451–460 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#451

Earlier quoted context omitted.

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

>the previous model retroactively becomes total dogshit the moment a new one is released

Keep writing your code manually, nobody cares.

Re: Measuring the impact of AI on experienced open-source developer productivity

#452

Earlier quoted context omitted.

No one is claiming 10x perf gains in vim. It’s just a fun geeky thing to use with a lot of zany customizations. And after two hellish years of memory muscling enough keyboard bindings to finally be productive, you earned it! It’s a badge of pride! But we all know you’re still fat fingering ggdG on occasion and silently cursing to yourself.

> No one is claiming 10x perf gains in vim. Sure they are - or at least were, unitl the last couple years. Same thing with Emacs. It's hard to claim this now, because the entire industry shifted towards webshit and cloud-based practices across the board, and the classical editors just can't keep up with VS Code. Despite the latter introducing LSP, which leveled the playing field wrt. code intelligence itself, the sur…

> vim and Emacs are not equipped to handle.

You clearly don't have a slightest idea of what you're talking about.

Emacs is actually still amazing in the LLM era. Language is all about plain text. Plain text remains crucial and will remain important because it's human-readable, machine-parsable, version-control friendly, lightweight and fast, platform-independent, and resistant to obsolescence. Even when analyzing huge amounts of complex data - images, videos, audio-recordings, etc., we often have to reduce it to text representation.

And there's simply no tool better than Emacs today that is well-suited for dealing with plain text. Nothing even comes close to what you can do with text in Emacs.

Like, check this out - I am right now transcribing my audio notes into .srt (subtitle) files. There's subed-mode where you can read through subtitles, and even play the audio, karaoke style, while following the text. I can do so many different things from here - extract the summaries, search through things, gather analytics - e.g., how often have I said 'fuck' on Wednesdays, etc.

I can similarly play YouTube videos in mpv, while controlling the playback, volume, speed, etc. from Emacs; I can extract subtitles for a given video and search through them, play the vid from the exact place in the subs.

I very often grab a selected region of screen during Zoom sessions to OCR and extract text within it and put it in my notes - yes, I do it in Emacs.

I can probably examine images, analyze their elements, create comprehensive summaries, and formulate expert artistic evaluation and critique and even ask Emacs to read it aloud back to me - the possibilities are virtually limitless.

It allows you to engage with vast array of LLM models from anywhere. I can ask a question in the midst of typing a Slack reply or reading HN comments or when composing a git commit; I can fact-check my own assumptions. I can also use tools to analyze and refactor existing codebases and vibe-code new stuff.

Anything like that even five years ago seemed like a dream; today it is possible. We can now reduce any complex digital data to plain text. And that feels miraculous.

If anything, the LLM era has made Emacs an extremely compelling choice. To be honest, for me - it's not even a choice, it's the only seriously viable option I have - despite all its drawbacks. Everything else doesn't even come close - other options either lacking critical features or have merely promising ones. Emacs is absolutely, hands-down, one of the best tools we humans have ever produced to deal with plain text. Anyone who thinks it's an opinion and not a fact simply hasn't grokked Emacs or has no clue what you can do with it.

Re: Measuring the impact of AI on experienced open-source developer productivity

#453

Earlier quoted context omitted.

Invoking personality is to the behavioral science as invoking God is to the natural sciences. One can explain anything by appealing to personality, and as such it explains nothing. Psychologists have been trying to make sense of personality for over a century without much success (the best efforts so far have been a five factor model [Big 5] which has ultimately pretty minor predictive value), which is why most behav…

Behaviorism is a relic of the 1950s

Not really a relic. Reinforcement learning is one of the best model for learned behavior we have. In the 1950s however cognitive science didn’t exist, and behavioralists thought they could explain much more with their model than they could, so they oversold the idea, by a lot.

Cognitive science was able to explain stuff like biases, pattern recognition, language, etc. which behavioral science thought they could explain, but couldn’t. In the 1950s it was really the only game in town (except for psychometrics which failed in a way much more complete—albeit less spectacular—way then behaviorism), so understandably scientists (and philosophers) went a little overboard with it (kind of like evolutionary biology did in the 1920s).

I think a more fair viewpoint is to claim that behaviorism’s heyday in the 1950s has passed, but it still provides an excellent theoretical framework for some of human behavior, and along with cognitive science, is able to explain most of what we know about human behavior.

Re: Measuring the impact of AI on experienced open-source developer productivity

#454

Earlier quoted context omitted.

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

What if agentic coding results in _less_ dopamine than manual coding? Because honestly I think that's more likely and jives with my experience. There's no flow state to be achieved with AI tools (at the moment)

With manual coding, the big dopamine hit comes at the end of a task - that's your internal feeling of reward for completing something.

I would think this could contrast with agentic coding, where the AI keeps generating code, and then you iterate on this process to get the AI to fix its mistakes. With normal human code review, it takes longer to get revisions and can feel like a slog. But with AI that's a much tighter loop, so maybe developers feel extra productive from all these dopamine hits from each interaction with the agent.

When manually coding and in flow state I'd think it's a more consistent level of arousal, less spiky. Probably varies by person and coding style though, which might also explain why some people love TDD and others can't stand it?

Re: Measuring the impact of AI on experienced open-source developer productivity

#455

Earlier quoted context omitted.

Yeah I’m in my 40s and have been using vim for decades. Sure there was an occasional rando stirring up the forums about made-up productivity gains to get some traffic to their blog, but that was it. There has always been push back from many of the strongest vim advocates that the appeal is not about typing speed or whatever it was they were claiming. It’s just ergonomics and power. It’s just not comparable to the LLM…

I am really curious what your thoughts on zed are, given that it has a lot of features and is still mostly vim compatible (from what i know) so you have the same ergonomics and power and it has some sane defaults / I don't need to tinker as much with zed as I would have to with nvim. Its not that I don't like tinkering. I really enjoy tinkering with config files but I never could understand nvim personally since I us…

I haven’t tried zed and I’m getting old and set in my ways. If it ain’t broke don’t fix it and all that.

So if the claim is that I can get everything I have out of vim, most importantly being unbeatably fast text buffers, and I don’t need a suitcase full of config files, that’s very compelling.

Is that the promise of zed?

Re: Measuring the impact of AI on experienced open-source developer productivity

#456
post #434

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

Hmmm, I don't see it? Are debuggers hard to use? Sometimes. But the debugger is allowing you to do something you couldn't actually do before. i.e. set breakpoints, and step through your code. So, while tricky to use, you are still in a better position than not having it. Just because you can get better at using something doesn't automatically mean that using it as a beginner makes you worse off. Same can be said for…

i guarantee you there were millions of people that needed to be forced to use excel because they thought they could do the calculations faster by hand.

we retroactively assume that everyone just obviously adopts new technology, yet im sure there were tons and tons of people that retired rather than learning how computers worked when the PC revolution was happening.

Re: Measuring the impact of AI on experienced open-source developer productivity

#457

Earlier quoted context omitted.

> No one is claiming 10x perf gains in vim. Sure they are - or at least were, unitl the last couple years. Same thing with Emacs. It's hard to claim this now, because the entire industry shifted towards webshit and cloud-based practices across the board, and the classical editors just can't keep up with VS Code. Despite the latter introducing LSP, which leveled the playing field wrt. code intelligence itself, the sur…

> vim and Emacs are not equipped to handle. You clearly don't have a slightest idea of what you're talking about. Emacs is actually still amazing in the LLM era. Language is all about plain text. Plain text remains crucial and will remain important because it's human-readable, machine-parsable, version-control friendly, lightweight and fast, platform-independent, and resistant to obsolescence. Even when analyzing hug…

At first I thought you were replying to me and this was a revival of the old vim + emacs wars.

I’m so glad we’re past that now and can join forces against a common enemy.

Thank you brother.

Re: Measuring the impact of AI on experienced open-source developer productivity

#458
post #246

What is interesting here is that all predictions were positive, but results are negative. This shows that everyone in the study (economic experts, ML experts and even developers themselves, even after getting experience) are novices if we look at them from the Dunning-Kruger effect [1] perspective. [1] https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect "The Dunning–Kruger effect is a cognitive bias in which…

> "The Dunning–Kruger effect is a cognitive bias in which people with limited competence in a particular domain overestimate their abilities."

No, they underestimated their own abilities for the most part; the estimates for AI-disallowed tasks were all undershot in terms of real implementation time.

What they overestimated was the ability of LLMs to provide real productivity gains on a given task.

Re: Measuring the impact of AI on experienced open-source developer productivity

#459

> The developers estimated how long it would take them to complete each task (a) under normal conditions Ah, there’s your issue. There’s not a developer in human history who hasn’t drastically underestimated how long it would take to complete a task.

On average the developers overestimated how long tasks would take when not using AI; they undershot their estimates on average. The opposite happened with AI-assisted tasks.

The conclusion isn't that "estimates are hard" (they can be), but rather that AI-assistance can lead people to believe they're being more productive than they actually are, because they incorrectly think they've spent less time.

The graphs in the paper tell part of that story; the time that is being reduced is in actual programming time, "Reading & Searching", "Testing & Debugging", but that time is being spent elsewhere, notably in parts specific to LLMs (reviewing output, prompting, waiting for the AI to spit out results).

Re: Measuring the impact of AI on experienced open-source developer productivity

#460

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

On the other hand if you don't use vim, emacs, and other spawns from hell, you get labeled a noob and nothing can ever be said about their terrible UX. I think we can be more open minded that an absolutely brand new technology (literally did not exist 3y ago) might require some amount of learning and adjusting, even for people who see themselves as an Einstein if only they wished to apply themselves.

> you get labeled a noob

No one would call one a noob for not using Vim or Emacs. But they might for a different reason.

If someone blindly rejects even the notion of these tools without attempting to understand the underlying ideas behind them, that certainly suggests the dilettante nature of the person making the argument.

The idea of vim-motions is a beautiful, elegant, pragmatic model. Thinking that it is somehow outdated is a misapprehension. It is timeless just like musical notation - similarly it provides compositional grammar and universal language, and leads to developing muscle memory; and just like it, it can be intimidating but rewarding.

Emacs is grounded on another amazing idea - one of the greatest ideas in computer science, the idea of Lisp. And Lisp is just as everlasting, like math notation or molecular formulas — it has rigid structural rules and uniform syntax, there's compositional clarity, meta-reasoning and universal readability.

These tools remain in use today despite the abundance of "brand new technology" because time and again these concepts have proven to be highly practical. Nothing prevents vim from being integrated into new tools, and the flexibility of Lisp allows for seamless integration of new tools within the old-school engine.

Post reply on HN