Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Productivity gains from AI coding assistants haven’t budged past 10% – survey

shiftmag.dev

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#71

Post by lbreakjai »

qudat wrote: post #31
overgard wrote: post #5

You're only as fast as your biggest bottleneck. Adding AI to an existing organization is just going to show you where your bottlenecks are, it's not going to magically make them go away. For most companies, the speed of writing code probably wasn't the bottleneck in the first place.

Agreed. The bottleneck is QA/Code review and that is never going away from most corps. I've never worked at a job in tech that didn't require code review and no, asking a code agent to review a PR is never going to be "good enough". And here we are, the central argument for why code agents are not these job killing hype beasts that are so regularly claimed. Has anyone seen what multi-agent code workflows produce? Tak…

My head of engineering spent half a day creating a complex setup of agents in opencode, to refactor a data model across multiple repositories. After a day running agents and switching between providers to work around the token limits, it dumped a -20k +30k change set we'll need to review.

If we're very lucky, we'll break even time wise compared to just running a single agent on a tight leash.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#72

Post by lunar_mycroft »

samuelknight wrote: post #60
jdlshore wrote: post #32

This is self-reported productivity, in that devs are saying AI saves them about 4 hours per week. But let’s not forget the METR study that found a 20% increase in self-reported productivity but a 19% decrease in actual measured productivity. (It used a clever and rigorous technique for measuring productivity differences, BTW, for anyone as skeptical of productivity measures as I am.)

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... That info is from mid 2025, talking about models released in Oct 2024 and Feb 2025. It predates tools like Claude Code and Codex, Lovable was 1/3 current ARR, etc. This might still be true but we desperately need new data.

None of those changes address the issue jdlshore is pointing out: self assessed developers productivity increases from LLMs are not a reliable indication of actual productivity increases. It's true that modern LLMs might have less of a negative impact on productivity or increase it, but you won't be able to tell by asking developers if they feel more productive.

(Also, Anthropic released Claude Code in Febuary of 2025, which was near the start of the period the study ran).

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#73

Post by keeda »

The real takeaway here -- also corroborated by the DORA 2025 report https://dora.dev/research/2025/ -- is that more than anything, AI amplifies your current development culture. Organizations with strong quality control discipline enjoy more velocity, those with weak practices suffer more outages.

Expecting AI to magically overcome your development culture is like expecting consultants to magically fix your business culture.

Furthermore, by various estimates, engineers only spend 10 - 60% of their time on actual code. So, given that currently AI is largely used only for coding activities, 10% is actually considerable savings.

Also this is the result of retro-fitting AI into existing workflows; actual "AI-native" workflows would probably look very different, likely having refactored in other parts of software engineering. Spotify's "Honk" workflow is probably just a starting point.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#74

Post by monkaiju »

samuelknight wrote: post #60
jdlshore wrote: post #32

This is self-reported productivity, in that devs are saying AI saves them about 4 hours per week. But let’s not forget the METR study that found a 20% increase in self-reported productivity but a 19% decrease in actual measured productivity. (It used a clever and rigorous technique for measuring productivity differences, BTW, for anyone as skeptical of productivity measures as I am.)

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... That info is from mid 2025, talking about models released in Oct 2024 and Feb 2025. It predates tools like Claude Code and Codex, Lovable was 1/3 current ARR, etc. This might still be true but we desperately need new data.

Yeah new data would be great, but i feel like these tools are not substantively better and this is becoming the new "its different this time!"

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#75

Post by Yodel0914 »

co_king_5 wrote: post #34
qudat wrote: post #31

Earlier quoted context omitted.

Agreed. The bottleneck is QA/Code review and that is never going away from most corps. I've never worked at a job in tech that didn't require code review and no, asking a code agent to review a PR is never going to be "good enough". And here we are, the central argument for why code agents are not these job killing hype beasts that are so regularly claimed. Has anyone seen what multi-agent code workflows produce? Tak…

[dead]

That’s… not how software works, no matter how it is produced. Complexity is the enemy; always.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#76

Post by lunar_mycroft »

williamcotton wrote: post #57
jdlshore wrote: post #32

This is self-reported productivity, in that devs are saying AI saves them about 4 hours per week. But let’s not forget the METR study that found a 20% increase in self-reported productivity but a 19% decrease in actual measured productivity. (It used a clever and rigorous technique for measuring productivity differences, BTW, for anyone as skeptical of productivity measures as I am.)

Has the METR study been replicated?

Not a scientific study, but someone did replicate the experiment on themselves [0] and found that in their case, any effect from LLM use wasn't detectable in their sample. Notably they almost certainly had more experience with LLMs than most of the METR participants did.

[0] https://mikelovesrobots.substack.com/p/wheres-the-shovelware...

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#77

Post by oblio »

MYEUHD wrote: post #61
oblio wrote: post #44

Earlier quoted context omitted.

> Take a look at openclaw, the code base is an absolute disaster. 500k LoC for something that can be accomplished in 10k. Mission accomplished: acquhire worth probably millions and millions. I agree with you, by the way.

It was a hire not an acquihire. There was no acquisition.

There was a big payoff on signing so to-may-to, to-mah-to.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#78

Post by orwin »

keeda wrote: post #73

The real takeaway here -- also corroborated by the DORA 2025 report https://dora.dev/research/2025/ -- is that more than anything, AI amplifies your current development culture. Organizations with strong quality control discipline enjoy more velocity, those with weak practices suffer more outages. Expecting AI to magically overcome your development culture is like expecting consultants to magically fix your business…

I'm pretty sure it has to do with the individual as well as the culture. Juniors/new hire use AI to multiply by two their wrong/unsafe output, and seniors then have to spend more time correcting it.

I'll be honest: I piss poor code, each time I come back to an old project I see where I could have done better. New hires are worse, but before AI (and especially Opus) they didn't produce that much code before spending like 6 months learning (I'm on a netsec tooling team). Now, they start producing code after two weeks or less, and every line have to be checked because they don't understand what they are doing.

I think my personal output was increased by 15% on average (maybe 5 on difficult projects), but our team output decreased overall.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#79

Post by moralestapia »

emp17344 wrote: post #65
randomtoast wrote: post #35

Earlier quoted context omitted.

This will lead to natural selection. As AI becomes increasingly integrated into all areas, companies that manage it less effectively than others will face greater selection pressure.

Or, AI will turn out to just not be that useful.

It's such a weird effect.

At a personal level, AI has made non-trivial improvements to my life. I can clearly see the value in there.

At an organizational level, it tends to get in the way much more than helping out. I do not yet see the value in there.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#80

Post by keeda »

jdlshore wrote: post #32

This is self-reported productivity, in that devs are saying AI saves them about 4 hours per week. But let’s not forget the METR study that found a 20% increase in self-reported productivity but a 19% decrease in actual measured productivity. (It used a clever and rigorous technique for measuring productivity differences, BTW, for anyone as skeptical of productivity measures as I am.)

Let's also not forget the multiple other studies that found significant boosts to productivity using rigorous methods like RCTs.

However, because these threads always go the same way whenever I post this, I'll link to a previous thread in hopes of preempting the same comments and advancing the discussion! https://news.ycombinator.com/item?id=46559254

Also, DX (whose CTO was giving the presentation) actually collects telemetry-based metrics (PR's etc.) as well: https://getdx.com/uploads/ai-measurement-framework.pdf

It's not clear from TFA if these savings are self-reported or from DX metrics.

Post reply on HN