Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

intrepidkarthi.com

Post reply on HN
Print view View original

51–60 of 115 posts

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#51

Post by dakolli »

loveparade wrote: post #10

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…

I have found llms to be utterly useless for frontend (tailwind included).

That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off.

I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "world", customer and product knowledge is required to design and implement great UX/UI?

I promise you are not going to be able to translate all this internalized understanding to an LLM and have it do your "tailwind integration" It actually sucks at all frontend outside of the 3 types of page layouts it understand.. Shitty landing pages, generic dashboards and shitty blog layouts.

Ya'll yearn for slop though so maybe everything will just become shit anyways.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#52

Post by torginus »

ianbutler wrote: post #13

2025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…

Either way, it's not a dramatic improvement. Thankfully I work in an environment where with little bureaucracy so my time is actually spent doing technical work.

I do think AI has been a huge boon to productivity in many ways, but looking at feature timelines, I think it's pretty clear the 'critical shortest path' of key features hasn't been sped up by that much.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#53

Post by loveparade »

dakolli wrote: post #51
loveparade wrote: post #10

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…

I have found llms to be utterly useless for frontend (tailwind included). That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off. I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "worl…

Fair point, I was more trying to make a statement about the amount of training data available, not the "difficulty" of the task. I just used Tailwind as an example because it is so ubiquitous with so much training data for LLMs to learn from, while any niche application doesn't have that.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#54

Post by [deleted] »

koe123 wrote: post #7
jiggawatts wrote: post #2

"19 August 2025" This may as well have been written in the stone ages, when we were banging AI rocks together. I just did a ~6 month project in ~2 weeks using a frontier model. I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!

Personal or work? Used by anyone or for fun?

[deleted]

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#55

Post by dakolli »

shawabawa3 wrote: post #9
shaky-carrousel wrote: post #6

There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.

I'm convinced this is what causes people to feel productive with vim

people that use vim motions/shortcuts/keyboard workflows are more productive, this is undeniable..

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#56

Post by onion2k »

hnlmorg wrote: post #22
onion2k wrote: post #15

Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to…

I wouldn’t even trust experienced developers to merge code without peer review.

I would, but that's mostly because I don't trust PRs to catch real problems. Someone reviewing the changes in a codebase is never going to spot an architectural or code design issue, and those are the real ones you need to care about. In my experience 95% of everything that's caught by a human PR review could have been caught by a linter or a formatter before the PR was opened.

If you trust your team to care about quality then PRs aren't necessary, and if you don't then why are you trusting them to catching problems in PR reviews?

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#57

Post by nikau »

js8 wrote: post #50
ianbutler wrote: post #13

2025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…

They also say "Wider adoption of AI has made it more difficult to measure task-level productivity" I think there is a simple reason for that. If you automate something, you make the measureable/predictable thing faster. So the hard to measure/predict part of the job will take more share of the time, and overall difficulty to measure/predict goes up. I think this is what happened with Agile Scrum - as developers becam…

I've held this stance on agile for a long time - it coincided with mainstream adoption of ssds, windows with memory protection and google search - all of which sped delivery despite agile, not because of agile.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#58

Post by lars512 »

verzali wrote: post #47
ianbutler wrote: post #13

2025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…

That post literally says the results are unreliable...

...and in particular it says that one of the reasons is that developers are refusing to participate in the non-AI branch, and when they do, changing what tasks they select to those where AI would be less useful.

Overall this suggests to them that the current speedup is likely greater than what the study could measure.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#59

Post by titanomachy »

jdkoeck wrote: post #24

> The honest counter, and it matters here more than usual. This is most likely the dip in a J-curve, not the destination. Oh, the irony of this post being AI-generated.

I don’t understand how these things still write so annoyingly. Eliminating just a handful of tells would make a big difference.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#60

Post by [deleted] »

ImprobableTruth wrote: post #31
jiggawatts wrote: post #2

"19 August 2025" This may as well have been written in the stone ages, when we were banging AI rocks together. I just did a ~6 month project in ~2 weeks using a frontier model. I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!

>I just did a ~6 month project in ~2 weeks using a frontier model. Claims like this are hard for me to take seriously because 'good' models have been available since the start of the year. So, if they really 10x one's productivity, then people should be able to have gotten done 5 years worth of work since then, but I've never actually seen anybody show any project like this.

[deleted]
Post reply on HN