Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

431–440 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#431

Post by saturneria »

card_zero wrote: post #213
jstummbillig wrote: post #194

Earlier quoted context omitted.

Convenient for whom and what...? There is nothing tangible to gain from you believing or not believing that someone else does (or does not) get a productivity boost from AI. This is not a religion and it's not crypto. The AI users' net worth is not tied to another ones use of or stance on AI (if anything, it's the opposite). More generally, the phenomenon this is quite simply explained and nothing surprising: New thi…

I saw that edit. Indeed you can't predict that rejecting a new thing is part of a routine of being wrong. It's true that "it's strange and new, therefore I hate it" is a very human (and adorable) instinct, but sometimes it's reasonable.

It is an even more human reaction when the new strange thing directly threatens to upend and massively change the industry that puts food on your table.

The steam-powered loom was not good for the luddites either. Good for society at large in the long term but all the negative points that a 40 year old knitter in 1810 could make against the steam-powered loom would have been perfectly reasonable and accurate judged on that individual's perspective.

Re: Measuring the impact of AI on experienced open-source developer productivity

#432

Post by xandrius »

ivanovm wrote: post #309
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

On the other hand if you don't use vim, emacs, and other spawns from hell, you get labeled a noob and nothing can ever be said about their terrible UX.

I think we can be more open minded that an absolutely brand new technology (literally did not exist 3y ago) might require some amount of learning and adjusting, even for people who see themselves as an Einstein if only they wished to apply themselves.

Re: Measuring the impact of AI on experienced open-source developer productivity

#433

Post by jahnu »

reverendsteveii wrote: post #31
noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

well we used to have a sort of inverse pareto where 80% of the work took 80% of the effort and the remaining 20% of the work also took 80% of the effort. I do think you're onto something with getting pebbles out of the road inasmuch as once I know what I need to do AI coding makes the doing much faster. Just yesterday I was playing around with removing things from a List object using the Java streams API and I kept r…

> once I know what I need to do AI coding makes the doing much faster

Most commenters on this paper seem to not respond to the strongest result from it. That is, the developers wrongly thought and felt that using AI had sped up their work. So we need to be super cautious about what we think we know.

Re: Measuring the impact of AI on experienced open-source developer productivity

#434

Post by themk »

viraptor wrote: post #313
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

Hmmm, I don't see it? Are debuggers hard to use? Sometimes. But the debugger is allowing you to do something you couldn't actually do before. i.e. set breakpoints, and step through your code. So, while tricky to use, you are still in a better position than not having it. Just because you can get better at using something doesn't automatically mean that using it as a beginner makes you worse off.

Same can be said for version control and programming.

Re: Measuring the impact of AI on experienced open-source developer productivity

#435

Post by phyzome »

lmeyerov wrote: post #358
phyzome wrote: post #341

Earlier quoted context omitted.

Have you actually measured this? Because one of the big takeaways from this study is that people are bad at predicting and observing their own time spent.

yes, I keep prompt plan logs At the same time... that's not why I'm comfortable writing this. It's pretty obvious when you know what good vs bad feels like here and adjust accordingly: 1. Good: You are able to generate a long plan and that plan mostly works. These are big wins _as long as you are multitasking_: you are high throughput, even if the AI is slow. Think running 5-20min at a time for pretty good progress,…

Oh, that's not quite what I was asking about -- I was wondering if you've compared AI vs no-AI for tasks, and kept measurements of that. But it sounds like you're not in a position to do so.

Re: Measuring the impact of AI on experienced open-source developer productivity

#436

Post by otabdeveloper4 »

mattmanser wrote: post #202
steveklabnik wrote: post #185

Earlier quoted context omitted.

Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.

Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model c…

> but now I see its reasoning

It's not showing its reasoning. "Reasoning" models are trained to output more tokens in the hope that more tokens means less hallucinations.

It's just a marketing trick and there is no evidence this sort of fake ""reasoning"" actually gives any benefit.

Re: Measuring the impact of AI on experienced open-source developer productivity

#437

Post by lmeyerov »

phyzome wrote: post #435
lmeyerov wrote: post #358

Earlier quoted context omitted.

yes, I keep prompt plan logs At the same time... that's not why I'm comfortable writing this. It's pretty obvious when you know what good vs bad feels like here and adjust accordingly: 1. Good: You are able to generate a long plan and that plan mostly works. These are big wins _as long as you are multitasking_: you are high throughput, even if the AI is slow. Think running 5-20min at a time for pretty good progress,…

Oh, that's not quite what I was asking about -- I was wondering if you've compared AI vs no-AI for tasks, and kept measurements of that . But it sounds like you're not in a position to do so.

Gotcha - and yep, I did a small internal natural qualitative experiment with our AI head. It lines up with what I was writing around the benefit being throughput after 2-3 week investment:

* Setup: I did a whole cloth clean room rewrite at production-grade an MCP server that was previously prototyped by the AI head. I intentionally went all-in for this "first serious" agentic coding project: ultimately * Baseline: If both of us did manually, I had already estimated similar times to completion due to differences in task scope naturally matching differences in proficiency for those tasks. Despite being new to agentic coding at the time, key aspects were in my favor: senior dev, 2 years of ~daily prompt engineering experience (tactics), PhD in code synthesis (strategy), and the repo setup with various lint/type/test guardrails.

* Result: About the same 1-2 weeks as a junior vibes coder vs the manual coder

* Experience: It was clear the first 1-2 weeks were slow due to onboarding myself and the codebase to agentic coding. That first week was especially rough. While I could get long runs going during it, I was doing the more confidently week 2, and switching to figuring out how to do parallel agents. Near the end, I was switching to doing multiple long runs in parallel on different tasks, where I could maintain maybe 2-4, but managing more gets exhausting, and especially when any are shorter runs.

Separately, we have a genAI team and a GPU Python team both switching to agentic coding. The natural difference in prompt engineering experience across teams seems to have individuals on one team picking up faster than the other, when gauged by the ability to do long runs

The initial experiment is part of why I view current agentic coding being more about overall coding throughput, and not latency within any specific task. If someone can reliably trigger long runs, do multiple in parallel, and doesn't waste time in interactive chatting, the difference is stark.

Likewise, reinforced by both of the above cases, looking for throughput improvements in the first 2-3w seems aggressive. A lot of blogposts seem to be from people new to high-quality prompting, tackling repos/tasks not setup for agents, and as limited nights & weekend efforts. To get to clear throughput multipliers from multiple agents working in parallel on good long runs.. I'd expect that to hit at month 2 or 3 when someone isn't as all in as I was able to be. It's more like a skill with setup, so takes investment before you reap the rewards.

Re: Measuring the impact of AI on experienced open-source developer productivity

#438

Post by ycombinatornews »

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

Thank you for the last paragraph.

Same thought came when I was reading the article and glad I am not alone.

Anecdotally, most common productivity boost is coming from cutting down weird slow steps in processes. Write an automation script, campaign previewer for marketing, etc etc.

Coding seems to transform to be a more efficient (again anecdotally) but not entirely faster. You can do a better work on a new feature in the same or slightly smaller time.

Idle time at 4% was interesting. I think this number goes higher the more you use a specific tool and adjust your workflow to that

Re: Measuring the impact of AI on experienced open-source developer productivity

#439

Post by Imustaskforhelp »

fingerlocks wrote: post #410
TeMPOraL wrote: post #392

Earlier quoted context omitted.

> No one is claiming 10x perf gains in vim. Sure they are - or at least were, unitl the last couple years. Same thing with Emacs. It's hard to claim this now, because the entire industry shifted towards webshit and cloud-based practices across the board, and the classical editors just can't keep up with VS Code. Despite the latter introducing LSP, which leveled the playing field wrt. code intelligence itself, the sur…

Yeah I’m in my 40s and have been using vim for decades. Sure there was an occasional rando stirring up the forums about made-up productivity gains to get some traffic to their blog, but that was it. There has always been push back from many of the strongest vim advocates that the appeal is not about typing speed or whatever it was they were claiming. It’s just ergonomics and power. It’s just not comparable to the LLM…

I am really curious what your thoughts on zed are, given that it has a lot of features and is still mostly vim compatible (from what i know) so you have the same ergonomics and power and it has some sane defaults / I don't need to tinker as much with zed as I would have to with nvim.

Its not that I don't like tinkering. I really enjoy tinkering with config files but I never could understand nvim personally since I usually want a lsp / good enough experience that nvim or any lunarvim etc. couldn't provide without me installing additional software.

Re: Measuring the impact of AI on experienced open-source developer productivity

#440

Post by hopeless »

evanelias wrote: post #233
pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

What if agentic coding results in _less_ dopamine than manual coding? Because honestly I think that's more likely and jives with my experience.

There's no flow state to be achieved with AI tools (at the moment)

Post reply on HN