The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
1–10 of 115 posts
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#2"19 August 2025"
This may as well have been written in the stone ages, when we were banging AI rocks together.
I just did a ~6 month project in ~2 weeks using a frontier model.
I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#3"19 August 2025" This may as well have been written in the stone ages, when we were banging AI rocks together. I just did a ~6 month project in ~2 weeks using a frontier model. I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!
What was the project? Could you share the code?
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#4“16 developers across 246 tasks”
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#5This study was shown to be flawed at the time; METR has retracted it. And it doesn't take into account current frontier models.
AI makes you more productive. This is no longer up for debate. The energy you spend arguing last year's talking points is better spent knuckling down and learning the tools.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#6There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#7"19 August 2025" This may as well have been written in the stone ages, when we were banging AI rocks together. I just did a ~6 month project in ~2 weeks using a frontier model. I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!
Personal or work? Used by anyone or for fun?
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#8“16 developers across 246 tasks”
With barely any experience or training with AI.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#9There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.
I'm convinced this is what causes people to feel productive with vim
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#10These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a different distribution of tasks.
A frontend dev doing tailwind integration for his day job is gonna see very different speedups than someone working in a niche scientific codebase. Taking the average makes about as much sense as taking the average of the speedup from calculators for a mathematician, a farmer, and an elementary school student.