This study was shown to be flawed at the time; METR has retracted it. And it doesn't take into account current frontier models. AI makes you more productive. This is no longer up for debate. The energy you spend arguing last year's talking points is better spent knuckling down and learning the tools.
What tools? The ones that will be outmoded in 6 months, or the ones that will be banned in 6 months?
The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
71–80 of 115 posts
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#72Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#73Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#74Earlier quoted context omitted.
That post literally says the results are unreliable...
...and in particular it says that one of the reasons is that developers are refusing to participate in the non-AI branch, and when they do, changing what tasks they select to those where AI would be less useful. Overall this suggests to them that the current speedup is likely greater than what the study could measure.
Like, what people are saying is, “That old study was wrong! They did a new broken study that overturned it!”
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#75Earlier quoted context omitted.
I'm convinced this is what causes people to feel productive with vim
people that use vim motions/shortcuts/keyboard workflows are more productive, this is undeniable..
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#76My two bosses are anti-AI. Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors. People will always create a world narrative that matches what they already believe. Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it rei…
>feel Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#77So why are all the top labs using the tools internally? They are lying or they are stupid? Devs wish that was true but it isn't and it will get better.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#78There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.
I'm convinced this is what causes people to feel productive with vim
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#79My two bosses are anti-AI. Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors. People will always create a world narrative that matches what they already believe. Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it rei…
>feel Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#80These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…
I have found llms to be utterly useless for frontend (tailwind included). That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off. I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "worl…
I got very frustrated with LLMs and their inability to apply good taste or maintain consistent design languages, and put the project on ice. But I decided to double down on more tooling and learn as much about frontend as I could because I also realized that frontend itself - the problem domain, the engineering culture (or lack thereof), the historical baggage, the sheer size of the frontend api/language surface was part of the problem. And also there was/is a lack of good LLM and agent-oriented tooling that was a much deeper problem than I expected initially.
I originally thought I would just create skills/workflows and apis for generating sites from templates, but the problem is moreso that you need an entirely different kind of harness and development process for frontend, which doesn't really exist yet. Claude design is probably the most familiar gesture in that direction for most people but I think it's only scratching the surface. Our own "agentic playwright" is https://github.com/accretional/chromerpc/tree/main/chrome-pr... - IMO this kind of tool (both ours and Claude Design) is a major win for removing the largest, most frustrating frontend LLM painpoints (having a human doing QA and prodding the model to fix obviously-wrong outputs).
But the bigger problem is that the webdev tooling ecosystem is FUCKING AWFUL, and there are too many different ways to do something even using the actual base browser apis, let alone all the random ass low-quality tools and cargoculting that seeps into the models' way of working and thinking. That's not to say that tools like React are bad, necessarily, but that there is so much pre-LLM slop and churn and low quality/inconsistent work in the frontend ecosystem that you really need to be MUCH more knowledgable about the way browsers and the web actually work than the median frontend developer (especially the ones participating in the endless hype flavor of the months, generating all the noise that defines the engineering culture) to effectively use them. Or even better, if you know enough you can also NOT reach for them because you're able to just implement it via raw html/css/browser primitives instead of through 2000 node packages.
To be clear, I'm not saying frontend development is slop, but that it has a very high skill ceiling and requires a lot of very particular/thorny knowledge to be good at. I think the reason AI frontend looks so much like slop is that it hasn't been RLed against the actual web-standards in a way that lets it learn how to actually build good sites, it just has the median frontend engineer archetype from its pretraining and then some kind of RLVR to get it to produce workable, not-fucked-up code (the 3x3 grid, the slop hero, the unnecessary blinking green buttons, etc.). And also, for LLMs, maybe engaging with the webdev tool ecosystem beyond the core infrastructure layer and base apis/languages is more trouble than it's worth, because they often optimize for "I want a particular kind of UX and don't know how to implement it directly, but I do know how to find a package and call it, then prod it into working".
LLMs need something more like a browser-harness, a meta-design system, per-design-language component management tooling, and a non-slop build system. They also generally need much better support/more sophisticated UX for hierarchical iframes, CSP, etc. which is a space that is not very well-explored despite its potential, because most frontend devs find it too hard or complicated.
People are already starting to build these and I think we'll get there in the next year. The hardest piece of the puzzle is figuring out how to structure RL training envs to learn frontend directly against web standards, because web standards are very complex and high-surface; but this is also the most promising because it's how you get Mythos-like superhuman performance. We have a project to build some of the base domain modeling/search tooling needed for frontend RL, eg https://github.com/accretional/proto-css, but it's early days. You should definitely try agentic browser tooling if you haven't yet because it makes a huge difference in getting existing LLMs to be more effective at frontend, and automating most of the debugging. It's what allows us to eg fully automate creating gifs of models interacting with our site in the context of a user journey when we run tests: https://github.com/accretional/proto-css/tree/main/chrome-te...