Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

471–480 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#471

Post by iLemming »

xandrius wrote: post #470
iLemming wrote: post #460

Earlier quoted context omitted.

> you get labeled a noob No one would call one a noob for not using Vim or Emacs. But they might for a different reason. If someone blindly rejects even the notion of these tools without attempting to understand the underlying ideas behind them, that certainly suggests the dilettante nature of the person making the argument. The idea of vim-motions is a beautiful, elegant, pragmatic model. Thinking that it is somehow…

One could try to be poetic with LLMs in order to make their point stronger and still convince absolutely no one who wasn't already convinced. I'm sure nobody really reject the notion of LLMs but sure as hell do like to moan if the new technology doesn't absolutely perfect fit their own way of working. Does that make them any different than people wanting an editor which is intuitive to use? Nobody will ever know.

> still convince absolutely no one who wasn't already convinced.

I don't know, people change their opinions all the time. I wasn't convinced about many ideas throughout my career, but I'm glad I found convincing arguments for some of them later.

> wanting an editor which is intuitive to use

Are you implying that Vim and Emacs are not?

Intuitive != Familiar. What feels unintuitive is often just unfamiliar. Vim's model actually feels pretty intuitive after the initial introduction. Emacs is pretty intuitive for someone who grokked Lisp basics - structural editing and REPL-driven development. The point is also subjective, for some people "intuitive editor" means "works like MS Word", but that's just one design philosophy, not an objective standard.

Tools that survive 30+ years and maintain passionate user bases must be doing something right, no?

> the new technology doesn't absolutely perfect fit their own way of working.

Emacs is extremely flexible, and thanks to that, I've rarely complained about new things not fitting my ways. I bend tools to fit my workflow if they don't align naturally — that's just the normal approach for a programmer.

Re: Measuring the impact of AI on experienced open-source developer productivity

#472

Post by ternaus »

I guess, I am experienced open-source developer

(https://github.com/albumentations-team/Albumentations)

15k stars, 5 million monthly downloads

----

It may happen that Cursor in the agentic mode writes code slower than I am. But!

It frees me from being in the IDE 100% of the time.

There is infinite list of educational videos, blog posts, scientific papers, hacker news, twitter, reddit that I want to read and going through them, while agents do their job is ultra convenient.

=> If I think about "productivity" in a broader way => with Cursor + agents, my overall productivity moved to a whole another level.

Re: Measuring the impact of AI on experienced open-source developer productivity

#473

Post by sanderjd »

fingerlocks wrote: post #377
sanderjd wrote: post #328

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Maybe, but it isn't hard to think of developer tools where this is the case. This is the entire history of editor and IDE wars. Imagine running this same study design with vim. How well would you expect the not-previously-experienced developers to perform in such a study?

No one is claiming 10x perf gains in vim. It’s just a fun geeky thing to use with a lot of zany customizations. And after two hellish years of memory muscling enough keyboard bindings to finally be productive, you earned it! It’s a badge of pride! But we all know you’re still fat fingering ggdG on occasion and silently cursing to yourself.

Huh? Most people use tools like vim for productivity...

I agree with you that AI dev tools are overhyped at the moment. But IDEs were, in fact, overhyped (to a lesser degree) in the past.

Re: Measuring the impact of AI on experienced open-source developer productivity

#474

Post by sanderjd »

oytis wrote: post #398
sanderjd wrote: post #328

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Maybe, but it isn't hard to think of developer tools where this is the case. This is the entire history of editor and IDE wars. Imagine running this same study design with vim. How well would you expect the not-previously-experienced developers to perform in such a study?

What I like about IDE wars is that it remained a dispute between engineers. Some engineers like fancy pants IDEs and use them, some are good with vim and stick with that. No one ever assumed that Jetbrains autocomplete is going to replace me or that I am outdated for not using it - even if there might be a productivity cost associated with that choice.

Excellent point. But I do think that forcing people to use IDEs for productivity was a thing for awhile. But still agree that the current moment is a difference in kind not just in scale.

Re: Measuring the impact of AI on experienced open-source developer productivity

#475

Post by graeme »

narush wrote: post #112
tcdent wrote: post #16

This study neglects to incorporate the fact that I have forgotten how to write code.

Honestly, this is a fair point -- and speaks the difficulty of figuring out the right baseline to measure against here! If we studied folks with _no_ AI experience, then we might underestimate speedup, as these folks are learning tools (see a discussion of learning effects in section (C.2.7) - Below-average use of AI tools - in the paper). If we studied folks with _only_ AI experience, then we might overestimate spee…

>If we studied folks with _only_ AI experience, then we might overestimate speedup, as perhaps these folks can't really program without AI at all.

Wouldn't this be an underestimate, since without ai there'd be no forward progress at all? So ai-assisted is infinite speedup if the outputs are good.

Re: Measuring the impact of AI on experienced open-source developer productivity

#476

Post by player1234 »

lackoftactics wrote: post #466
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

Stay tuned, a new study is coming with another revelation: you aren't getting faster by using Vim when you are learning it. My previous employer didn't even allow me to use Vim until I learned it properly so it wouldn't affect my productivity. Why would using a cursor automatically make you better at something if it's just new to you and you are already an elite programmer according to this study?

How did you measure this? Was the conslusion of your studies that typing/editing speed was the real bottlekneck for a SWE becoming 10x?

Re: Measuring the impact of AI on experienced open-source developer productivity

#477

Post by player1234 »

novaleaf wrote: post #447
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

I've spent the last 2 months trying to figure out how to utilize AI properly, and only in the last week do I feel that I've hit upon a workflow that's actually a force multiplier (vs divisor).

[dead]

Re: Measuring the impact of AI on experienced open-source developer productivity

#478

Post by player1234 »

itsoktocry wrote: post #451
blibble wrote: post #169

Earlier quoted context omitted.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

> the previous model retroactively becomes total dogshit the moment a new one is released Keep writing your code manually, nobody cares.

And nobody will notice.

Re: Measuring the impact of AI on experienced open-source developer productivity

#479

Post by exiguus »

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

> My intuition here is that this study mainly demonstrated that the learning curve on AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learning curve.

Could be the case for some, but I also think, that there is not much to climb on the learning curve for AI agents.

In my opinion, its more interesting, that the study also states, that AI capabilities may be comparatively lower on existing code:

> Our results also suggest that AI capabilities may be comparatively lower in settings with very high quality standards, or with many implicit requirements (e.g. relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn.

This is consistent with my personal/pear experience. On existing code: You have to do try and error with AI until you get a 'good' result. Or highly modify AI generated code by yourself (which is often slower then writing it yourself from the beginning).

Re: Measuring the impact of AI on experienced open-source developer productivity

#480

Post by sarchertech »

atleastoptimal wrote: post #310
artee_49 wrote: post #128

Earlier quoted context omitted.

Even for senior levels the claim has been that AI will speed up their coding (take it over) so they can focus on higher level decisions and abstract level concepts. These contributions are not those and based on prior predictions the productivity should have gone up.

It would be different I'm sure if they were making contributions to repos they had less familiarity with. In my experience and talking with those who use AI most effectively, it is best leveraged as a way of getting up to speed or creating code for a framework/project you have less familiarity with. The general ratio determining the effectiveness of non-AI coding vs AI coding is the familiarity the user has with the…

I’ve been doing this for a while now. Junior engineers are pretty near universally terrible when measured by short term ROI. The only reason you would ever want to pay a truly junior engineer is because you can teach them.

If someone told me “you can have a free junior engineer, but they get swapped out each week for a new person”, I’d say no thanks.

I’m sure someone could figure out a way to make money on in that situation, but it wouldn’t be by building anything I’d be comfortable attaching my name to or or would want to use myself

Post reply on HN