Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Productivity gains from AI coding assistants haven’t budged past 10% – survey

shiftmag.dev

Post reply on HN
Print view View original

91–99 of 99 posts

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#91

Post by ponector »

co_king_5 wrote: post #34
qudat wrote: post #31

Earlier quoted context omitted.

Agreed. The bottleneck is QA/Code review and that is never going away from most corps. I've never worked at a job in tech that didn't require code review and no, asking a code agent to review a PR is never going to be "good enough". And here we are, the central argument for why code agents are not these job killing hype beasts that are so regularly claimed. Has anyone seen what multi-agent code workflows produce? Tak…

[dead]

I imagine to get more robust code your agent can replace for loop with long lines of if-then statements. Later manager can brag about how many lines of code they created!

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#92

Post by jimbokun »

tyleo wrote: post #41
mytailorisrich wrote: post #10

Blunt opinion: Most devs are not that good and really only execute what they are told to do. The threat of AI for devs, and the way to drastically improve productivity is there: keep the better devs who can think systemically, who can design solutions, who can solve issues themselves and give them all the AI help available, cut the rest.

That’s how I feel too. When I was an architect at a ~300-person company, a big chunk of my job shifted to reviews, technical design docs, and guidance. I’m getting great results by feeding context like that into Claude Code, then reviewing and steering what it produces. It really does feel like a multiplier on me and I understand things enough to get my hands dirty where Claude struggles. Lately I’ve been wondering i…

I would collapse the senior engineer and architect roles into a single role and only hire people capable of doing both.

Owning the modules requires understanding the overall architecture and how the module’s API needs to fit into it, when a new module is required, when the architecture needs to change, and how to pragmatically migrate from one design to another in a live system.

All of that is easier if everyone is capable of operating at any level from overall architecture to micro optimizations.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#93

Post by zbentley »

co_king_5 wrote: post #34
qudat wrote: post #31

Earlier quoted context omitted.

Agreed. The bottleneck is QA/Code review and that is never going away from most corps. I've never worked at a job in tech that didn't require code review and no, asking a code agent to review a PR is never going to be "good enough". And here we are, the central argument for why code agents are not these job killing hype beasts that are so regularly claimed. Has anyone seen what multi-agent code workflows produce? Tak…

[dead]

I hope this is a troll.

The best, or cheapest, or most-loved, or most extensible software is often written to be as small as is practical in the problem.

Huge codebases are necessarily going to be harder to process by humans (LoC -> hours in the day) and LLMs (LoC -> tokens/context growth) alike.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#94

Post by JohnBooty »

jdlshore wrote: post #32

This is self-reported productivity, in that devs are saying AI saves them about 4 hours per week. But let’s not forget the METR study that found a 20% increase in self-reported productivity but a 19% decrease in actual measured productivity. (It used a clever and rigorous technique for measuring productivity differences, BTW, for anyone as skeptical of productivity measures as I am.)

    But let’s not forget the METR study that 
    found a 20% increase in self-reported productivity 
    but a 19% decrease in actual measured productivity.
Counting "time per PR" is as useless as counting lines of code.

An industry I think we spend ~10% of our time writing code and ~90% of our time maintaining it and building upon it.

The real metric is not "how long did that PR take" but "how much additional work will this PR create or save in the long run." -- ie did this create tech debt? Or did it actually save us a bunch of effort in the long run?

My experience with ChatGPT these last few years is that if used "conscientiously" it allows me to ship much higher quality code because it has been very good at finding edge cases and suggesting optimizations. I am quite certain that when viewed over the long haul it has been at least a 2X productivity gain, possibly even much more, because all those edge cases and perf issues it solved for me in the initial PR represent many hours of work that will never have to be performed in the future.

It is of course possible to use AI coding assistants in other ways, producing AI slop that passes tests but is poorly structured and understood.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#95

Post by ai_tools_daily »

10% feels right for the median case but wildly wrong for specific workflows.

Where I see massive gains (50%+ time savings): boilerplate generation, test writing, regex/config syntax I'd otherwise have to look up, and exploring unfamiliar codebases. These are high-frequency, low-creativity tasks where AI genuinely excels.

Where I see near-zero gains: debugging complex distributed systems, architectural decisions, understanding why code exists the way it does (not what it does), and navigating ambiguous requirements. These require context that doesn't fit in a prompt.

The 10% average likely masks a bimodal distribution. Developers who restructured their workflow around AI assistance probably see 30-40%. Developers who use it as a fancy autocomplete see 5%. Measuring "productivity" as a single number hides all the interesting variation.

Also worth noting: the biggest unlock isn't code generation — it's using AI to quickly evaluate multiple approaches before committing to one. That's hard to measure but extremely valuable.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#96

Post by edanm »

lunar_mycroft wrote: post #72
samuelknight wrote: post #60

Earlier quoted context omitted.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... That info is from mid 2025, talking about models released in Oct 2024 and Feb 2025. It predates tools like Claude Code and Codex, Lovable was 1/3 current ARR, etc. This might still be true but we desperately need new data.

None of those changes address the issue jdlshore is pointing out: self assessed developers productivity increases from LLMs are not a reliable indication of actual productivity increases. It's true that modern LLMs might have less of a negative impact on productivity or increase it, but you won't be able to tell by asking developers if they feel more productive. (Also, Anthropic released Claude Code in Febuary of 202…

> self assessed developers productivity increases from LLMs are not a reliable indication of actual productivity increases.

I believe the other direction makes more sense; if the studies disagree with self-reported information, it's more likely the studies are wrong. At the very least, it's worth heavily questioning whether the studies are wrong.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#97

Post by otabdeveloper4 »

8note wrote: post #59
otabdeveloper4 wrote: post #51

Earlier quoted context omitted.

You need human language programming-related questions to train on too, not just the code.

thats what the related chats are for?

And now you're training LLMs on LLM output.

No, you need something like Stackoverflow. The crowdsourced ratings system that Stackoverflow has (had?) is the crucial part.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#98

Post by MattPalmer1086 »

sshine wrote: post #63
otabdeveloper4 wrote: post #18

Earlier quoted context omitted.

> let's have LLMs check our code for correctness Lmao. Rofl even. (Testing is the one thing you would never outsource to AI.)

> Testing is the one thing you would never outsource to AI That's not really true. Making the AI write the code, the test, and the review of itself within the same session is YOLO. There's a ton of scaffolding in testing that can be easily automated. When I ask the AI to test, I typically provide a lot of equivalence classes. And the AI still surprises me with finding more. On the other hand, it's equally excellent a…

Yep. I have had had success getting AI to write tests. They all needed review, but still a massive speed up for me.

It made about 2 mistakes in over 100 tests, and the coverage of the tests was higher than I would have attempted.

So about 2 hours of work instead of 1 or 2 days of boring effort avoided and a better outcome.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#99

Post by christofosho »

lbreakjai wrote: post #71
qudat wrote: post #31

Earlier quoted context omitted.

Agreed. The bottleneck is QA/Code review and that is never going away from most corps. I've never worked at a job in tech that didn't require code review and no, asking a code agent to review a PR is never going to be "good enough". And here we are, the central argument for why code agents are not these job killing hype beasts that are so regularly claimed. Has anyone seen what multi-agent code workflows produce? Tak…

My head of engineering spent half a day creating a complex setup of agents in opencode, to refactor a data model across multiple repositories. After a day running agents and switching between providers to work around the token limits, it dumped a -20k +30k change set we'll need to review. If we're very lucky, we'll break even time wise compared to just running a single agent on a tight leash.

In our org that would not fly. They would be required to break it down. Did you or anyone tell them they need to make it readable for the rest of the team?
Post reply on HN