Live data from Hacker News

The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

intrepidkarthi.com

71–80 of 115 posts

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#71
post #5

This study was shown to be flawed at the time; METR has retracted it. And it doesn't take into account current frontier models. AI makes you more productive. This is no longer up for debate. The energy you spend arguing last year's talking points is better spent knuckling down and learning the tools.

What tools? The ones that will be outmoded in 6 months, or the ones that will be banned in 6 months?

Whatever tools come along, step one is getting into the mindset that typing code in is no longer part of your job. You are a designer and director, not a coder. As Steve Yegge says, if you still have an IDE open entering code by hand, you're one of the crappy engineers. You need to be getting into the habit of understanding the strengths and weaknesses of your model and agentic harness and using those to produce the results you want. When those change in six months, you adapt along with them. Adjusting to the new mindset is the biggest hurdle, and there'll be plenty of devs who can't, and won't make the cut or stay in the field for very long. Just like there were plenty of devs who couldn't adjust to anything beyond COBOL on punched cards.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#73
post #55

Earlier quoted context omitted.

I'm convinced this is what causes people to feel productive with vim

people that use vim motions/shortcuts/keyboard workflows are more productive, this is undeniable..

[dead]

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#74
post #58
post #47

Earlier quoted context omitted.

That post literally says the results are unreliable...

...and in particular it says that one of the reasons is that developers are refusing to participate in the non-AI branch, and when they do, changing what tasks they select to those where AI would be less useful. Overall this suggests to them that the current speedup is likely greater than what the study could measure.

It might suggest that but they can’t back it up because the study is broken and perhaps forever unrepeatable.

Like, what people are saying is, “That old study was wrong! They did a new broken study that overturned it!”

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#75
post #55

Earlier quoted context omitted.

I'm convinced this is what causes people to feel productive with vim

people that use vim motions/shortcuts/keyboard workflows are more productive, this is undeniable..

Vim makes some slow and incredibly tedious tasks, fast and efficient. Having said that, all those key-presses to switch modes do add up.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#76

My two bosses are anti-AI. Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors. People will always create a world narrative that matches what they already believe. Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it rei…

>feel Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist

Problem is, all the old metrics are now obsolete and nobody knows how to measure productivity anymore.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#78

There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.

I'm convinced this is what causes people to feel productive with vim

Do you have evidence to show they are not, in fact, more productive? It's easy to find things where a mouse is comically slow compared to vim. But any kind of religious adherence to tools isn't going to be the most efficient way. Like with just about anything you need to find a balance. An extremely large set of tools is just as bad as an extremely small set.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#79

My two bosses are anti-AI. Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors. People will always create a world narrative that matches what they already believe. Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it rei…

>feel Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist

It is however extremely hard to measure accurately with software engineering and every easy measurement immediately draws ire from devs here.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#80
post #51

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…

I have found llms to be utterly useless for frontend (tailwind included). That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off. I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "worl…

I ran into this with my first attempt at building a static site generator "for agents" last year. https://statue.dev

I got very frustrated with LLMs and their inability to apply good taste or maintain consistent design languages, and put the project on ice. But I decided to double down on more tooling and learn as much about frontend as I could because I also realized that frontend itself - the problem domain, the engineering culture (or lack thereof), the historical baggage, the sheer size of the frontend api/language surface was part of the problem. And also there was/is a lack of good LLM and agent-oriented tooling that was a much deeper problem than I expected initially.

I originally thought I would just create skills/workflows and apis for generating sites from templates, but the problem is moreso that you need an entirely different kind of harness and development process for frontend, which doesn't really exist yet. Claude design is probably the most familiar gesture in that direction for most people but I think it's only scratching the surface. Our own "agentic playwright" is https://github.com/accretional/chromerpc/tree/main/chrome-pr... - IMO this kind of tool (both ours and Claude Design) is a major win for removing the largest, most frustrating frontend LLM painpoints (having a human doing QA and prodding the model to fix obviously-wrong outputs).

But the bigger problem is that the webdev tooling ecosystem is FUCKING AWFUL, and there are too many different ways to do something even using the actual base browser apis, let alone all the random ass low-quality tools and cargoculting that seeps into the models' way of working and thinking. That's not to say that tools like React are bad, necessarily, but that there is so much pre-LLM slop and churn and low quality/inconsistent work in the frontend ecosystem that you really need to be MUCH more knowledgable about the way browsers and the web actually work than the median frontend developer (especially the ones participating in the endless hype flavor of the months, generating all the noise that defines the engineering culture) to effectively use them. Or even better, if you know enough you can also NOT reach for them because you're able to just implement it via raw html/css/browser primitives instead of through 2000 node packages.

To be clear, I'm not saying frontend development is slop, but that it has a very high skill ceiling and requires a lot of very particular/thorny knowledge to be good at. I think the reason AI frontend looks so much like slop is that it hasn't been RLed against the actual web-standards in a way that lets it learn how to actually build good sites, it just has the median frontend engineer archetype from its pretraining and then some kind of RLVR to get it to produce workable, not-fucked-up code (the 3x3 grid, the slop hero, the unnecessary blinking green buttons, etc.). And also, for LLMs, maybe engaging with the webdev tool ecosystem beyond the core infrastructure layer and base apis/languages is more trouble than it's worth, because they often optimize for "I want a particular kind of UX and don't know how to implement it directly, but I do know how to find a package and call it, then prod it into working".

LLMs need something more like a browser-harness, a meta-design system, per-design-language component management tooling, and a non-slop build system. They also generally need much better support/more sophisticated UX for hierarchical iframes, CSP, etc. which is a space that is not very well-explored despite its potential, because most frontend devs find it too hard or complicated.

People are already starting to build these and I think we'll get there in the next year. The hardest piece of the puzzle is figuring out how to structure RL training envs to learn frontend directly against web standards, because web standards are very complex and high-surface; but this is also the most promising because it's how you get Mythos-like superhuman performance. We have a project to build some of the base domain modeling/search tooling needed for frontend RL, eg https://github.com/accretional/proto-css, but it's early days. You should definitely try agentic browser tooling if you haven't yet because it makes a huge difference in getting existing LLMs to be more effective at frontend, and automating most of the debugging. It's what allows us to eg fully automate creating gifs of models interacting with our site in the context of a user journey when we run tests: https://github.com/accretional/proto-css/tree/main/chrome-te...

Post reply on HN