Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

intrepidkarthi.com

Post reply on HN
Print view View original

61–70 of 115 posts

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#61

Post by rob74 »

loveparade wrote: post #10

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…

To add to that, the only way to reliably measure speedup would be to give the same developer the same task twice, first without AI, then with AI, and the developer should have no previous knowledge (or the same level of previous knowledge) about the task each time he starts - which is inherently impossible. I didn't read the study, but from the article it looks like they compared the actual speed to prior estimates, and we all know how reliable those are?

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#62

Post by xg15 »

> AI speeds up typing, which was never the bottleneck for an expert in a codebase they already know.

For me as a dev, that's not the whole truth. Where I've found actual value in AI (and I think were some of that "perceived speedup" is coming from) is looking up things.

Unless you know the codebase and used libraries extremely well, you will have to do lots of "micro-lookups" during coding, where you have to find the specific APIs or library functions for your problem, then figure out how exactly you have to call them, how to handle the result, etc. That's lots of "research" work interleaved with actually writing the code.

AIs seem to be good enough to have a lot of that knowledge already baked into their weights, at least for popular platforms, so if you prompt it something, you can skip all that low-level lookup work or at least defer it until code review. Even during review, it's easier, because you don't have to come up with the appropriate library function from scratch, you only have to verify that the ones the AI used make sense and are used correctly.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#63

Post by onion2k »

alfalfasprout wrote: post #20
onion2k wrote: post #15

Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to…

small downsides like security holes? Those aren't small. Neither is creating a codebase that's an inextensible mess that even LLMs begin to struggle with. The reality is making good decisions and thinking about approaches take time. AI can absolutely make us faster at it but it's not magic and these speedups come with effort.

small downsides like security holes? Those aren't small

I'm British. I've been taught to turn understatement into an art.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#64

Post by [deleted] »

dakolli wrote: post #51
loveparade wrote: post #10

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…

I have found llms to be utterly useless for frontend (tailwind included). That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off. I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "worl…

[deleted]

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#65

Post by dofm »

ianbutler wrote: post #13

2025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…

Their followup study essentially says the followup study itself is possibly broken because developers will now not participate in some of the non-AI tasks and because the study pays less.

I would not, at all, suggest that this second study corrects or debunks the first.

Instead what it shows (if anything, i.e. if you can even put aside the regrettable choice to change the payment level, which affects applicant recruitment) is that the mindset shift has already happened: developers now don’t want to attempt some tasks without AI.

What that tells you is not (with any confidence at least) that they are faster, but perhaps that we are beyond the point that this can be meaningfully measured. AI could still be making developers slower, but developers aren’t going to be willing or perhaps able to help you find out.

Basically the job is different now.

What this does for me, perhaps, is vindicate my feelings. I can do agentic coding; I have learned the principles and some tools and I could learn more. But if this study is really reflective of how other developers feel now, I am done.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#67

Post by michaelt »

One thing I've noticed with generative AI is it's now easier than ever to write more lines of code.

Before, a backend guy asked to add an intranet page would make an austere page -bare html with barely any styling or javascript. Today, the same guy given the same task can turn in something with styling, javascript, internationalisation, interactive form validation, progress spinner, minification build stage, linting, maybe even automated browser tests.

And I have to code review it. Now the bottleneck of writing the code has been removed, I now find code review is the bottleneck - and a bottleneck facing much higher flow must either let more through, or start applying back pressure.

Sometimes I think an evil genie granted my wish for better tested code by trying to drown me in it.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#68

Post by Almondsetat »

dymk wrote: post #36
Almondsetat wrote: post #30

Earlier quoted context omitted.

>feel Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist

Performance review at FAANGs has always been vibes and soft skills.

And? That's not the point of discussion

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#69

Post by larodi »

So are you trying to imply that I've somehow accidentally stumbled upon more than 100k lines of new high-perf working code, done in less than 6 months, which is like not 20%, but 200% my actual output, and this code, already generating revenue for me and my employer, is something of the ordinary, and actually I can do 20% better typing it manually?

ROFL sorry

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#70

Post by Shitty-kitty »

jiggawatts wrote: post #2

"19 August 2025" This may as well have been written in the stone ages, when we were banging AI rocks together. I just did a ~6 month project in ~2 weeks using a frontier model. I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!

That's amazing. What's even more incredible is that somehow you managed to do a real code review and testing in that time-frame.
Post reply on HN