One thing I see missing in a lot of these discussions is whether or not the metric is solely based on speed. I think AI just allows you to look at your code in different ways and provides more chances to catch mistakes. I am definitely slower with AI assistance, but that is because I use it to increase the quality of my work.
The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
41–50 of 115 posts
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#422025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…
Even not touching the laughable sample size for both studies - almost halved sample size between 2025 and 2026? Sounds like a massive selection bias, and not in the way they're implying.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#432025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…
And to be specific, the METR study was using the Cursor harness with Claude Sonnet 3.5/3.7, along with other models of that era of the participant’s choosing.
Which is ancient at this point, and half a year older than the November 2025 inflection point when agentic coding got really good.
The original article is from August 2025, and the overall message to not trust ‘how it feels’ and rather measure outcomes seems right to me despite the outdated figures. On my team at least, we are seeing a noticeable inflection in work shipped with AI according to Weave.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#44Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#45My two bosses are anti-AI. Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors. People will always create a world narrative that matches what they already believe. Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it rei…
[deleted]
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#46If you linked to the actual source of the study [1] instead of a random blog only talks about the result, you would see the big banner that the authors put there noting that the study is horribly outdated. Current models do make developers faster.
[1] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#472025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…
That post literally says the results are unreliable...
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#48Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to…
I wouldn’t even trust experienced developers to merge code without peer review.
It is getting to the point where I wouldn't trust experienced developers to merge code without AI review. The latest generation of models are getting pretty good.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#49So why are all the top labs using the tools internally? They are lying or they are stupid?
Devs wish that was true but it isn't and it will get better.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#502025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…
They also say "Wider adoption of AI has made it more difficult to measure task-level productivity"
I think there is a simple reason for that. If you automate something, you make the measureable/predictable thing faster. So the hard to measure/predict part of the job will take more share of the time, and overall difficulty to measure/predict goes up.
I think this is what happened with Agile Scrum - as developers became more productive (for unrelated reasons, two main sources of SW developer productivity before AI were compilers and open source), the bureacracy (amount of meetings) increased, because the ratio of hard to measure vs easy to measure went up. Bureacracy is hard to measure, so it went up (as a share of work). I expect this only getting worse with more automation, such as AI. So I predict an increase in share of bureacracy compared to pre-AI world.
Either way, IMHO main point is automation has the opposite effect on human job predictability, it lowers it. Tasks we can easily automate are those that are easy to predict.