Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

intrepidkarthi.com

Post reply on HN
Print view View original

81–90 of 115 posts

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#81

Post by globular-toast »

michaelt wrote: post #67

One thing I've noticed with generative AI is it's now easier than ever to write more lines of code. Before, a backend guy asked to add an intranet page would make an austere page -bare html with barely any styling or javascript. Today, the same guy given the same task can turn in something with styling, javascript, internationalisation, interactive form validation, progress spinner, minification build stage, linting,…

I mean they could have turned in one of the countless HTML "boilerplate" projects before LLMs too. It hasn't been necessary to start completely from scratch for quite a while now. I'm surprised any professional web developer wouldn't maintain their own boilerplate as templates that they can quickly roll off the production line. Tools like cookiecutter[0] have been available for a long time. Sometimes I feel like LLMs are just allowing people to somewhat catch up to where others were decades ago.

[0] https://github.com/cookiecutter/cookiecutter

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#82

Post by Cthulhu_ »

loveparade wrote: post #10

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…

And even if the refactor is 100x faster (which is unlikely) - would that work have been done at all without AI? What would the effect be on total speed / througput if it was or was not done? These things get more and more difficult to measure.

What I'm really wondering is how much extra tasks are being done that wouldn't have been done without AI, and whether those actually have a payoff. That is, 100x faster versus not doing the work at all.

I'm hoping some enterprises that collect metrics on e.g. time-to-market, customer satisfaction, revenue, costs, etc will release an authoritative report some day.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#83

Post by beaugunderson »

onion2k wrote: post #15

Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to…

> Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now.

I built a UAT agent on top of claude-agent-sdk, it uses Playwright and can spin up a preview instance for PRs we open. It uses its knowledge of the code to create a test plan, runs that test plan, and takes screenshots as it does so. On a recent PR I made a change to our MFA implementation and assumed that I would need to test it myself since MFA requires setup (and a TOTP device!) but no, the agent drove Playwright to change the instance configuration to enable MFA, created a user with MFA enabled, wrote the tiniest (handful of lines) Python TOTP and used that as the device and successfully verified that MFA still worked as expected. I did not expect that ability, and I wrote it! It was also exhaustive in its probe of functionality, way more than I had planned to do, even though I had planned to do a lot given the critical nature of MFA.

Total token price for the UAT run: $5.69.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#84

Post by IanCal »

onion2k wrote: post #56
hnlmorg wrote: post #22

Earlier quoted context omitted.

I wouldn’t even trust experienced developers to merge code without peer review.

I would, but that's mostly because I don't trust PRs to catch real problems. Someone reviewing the changes in a codebase is never going to spot an architectural or code design issue, and those are the real ones you need to care about. In my experience 95% of everything that's caught by a human PR review could have been caught by a linter or a formatter before the PR was opened. If you trust your team to care about qu…

Outside of the benefit of some extra documentation around changes and having more than one person see what’s happened there are a few main safety parts here:

* It can be easier to see a problem in something that another person wrote, you’re not clouded by what you intended to write

* different skill sets mean two sets of eyes broadens the kind of problems that can be found (benefits from selecting good reviewers, maybe I’m tagged because I know the llm APIs and performances better by someone who has done refactoring to improve, say, internal caching that’s more their thing)

* just chance. There’s some chance you spot an issue, some chance someone else does. Combined it’s better.

* I disagree they can’t find architectural or design issues. You see repeated changes of the same kind, or tying together things that shouldn’t be, etc.

But yes, many things could be caught before opening. Lots I catch as I’m explaining the change, like rubber ducking. I quite like AI code reviews things for this, there’s a whole back and forth that can be avoided once you get past basic linting/test level things. Save the human time for understanding the higher level issues.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#85

Post by IanCal »

roenxi wrote: post #48
hnlmorg wrote: post #22

Earlier quoted context omitted.

I wouldn’t even trust experienced developers to merge code without peer review.

It is getting to the point where I wouldn't trust experienced developers to merge code without AI review. The latest generation of models are getting pretty good.

Definitely for the time and cost too.

If a dev costs $1k/day that’s $30 to spend 15 minutes looking at a pr. How much review does that in tokens get you? I’d wager you’d easily find lots of low level issues that are beyond basic linting with that reliably, and I feel like latest gen models can do really good work.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#86

Post by Cthulhu_ »

shaky-carrousel wrote: post #6

There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.

I'm sure this is the case; in both cases, the amount of "things happening" is higher (keystrokes, perceived actions per second, tokens used, words generated by LLM while it's "thinking", back and forth betwee LLM and user, etc) - but does it translate to measurable end results? tickets closed, features delivered, money earned vs cost (personnel + AI + hosting + other costs), etc.

As always this is hard to measure, or will take a while longer yet to draw conclusions from.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#88

Post by Avshalom »

100ms wrote: post #16
shawabawa3 wrote: post #9

Earlier quoted context omitted.

I'm convinced this is what causes people to feel productive with vim

Triggered by both of these comments.. interaction mode dictates a style of thinking. I have to use a mouse, I'm forced to use my eyes, which also means I probably have to use a massive screen. I have to pay attention to some hyperactive Intellisense-like feature, I'm forced to remove my attention from the problem. It's like saying you're convinced people reporting they feel more productive in a mauve-coloured room ar…

>> I'm forced to use my eyes, which also means I probably have to use a massive screen. I have to pay attention to some hyperactive Intellisense-like feature

What the hell are you talking about?

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#89

Post by Zababa »

loveparade wrote: post #10

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a differen…

>and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding

This isn't true in my experience, AI is great at gathering context through slack, repositories, emails, web pages. For building understanding too, provided you use it well.

Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

#90

Post by weitendorf »

shaky-carrousel wrote: post #6

There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.

There are multiple developer subcultures (nominally for productivity but mostly hobbyists) pretty much exclusively motivated by installing and configuring complex, visually-dense, high-learning-curve tooling and editor setups driven by the same psychology.

Besides the cultural association between keyboard navigation and complex tools with being a 1337 h4xx0r, I think there is something to be said about the process of tinkering with and learning how one's own tools work, or more generally experimenting with new, "interesting" ways of working than the default choice (which around where AI was at the time of this study), and being more engaged and thus more knowledgable about one's own work or problem domain, even if the overhead ends up being a poor investment time-wise upfront. Personally, if it took me 20% longer to accomplish something but I understood it 95%, vs 75% if I had done it the "fast way", I would almost always take the 20% latency hit, with the expectation that more knowledge/exposure to different tools and techniques would have much better ROI over time than marginally faster delivery.

There's a certain kind of developer (much moreso the kind all-in on AI in early 2025) who thinks that AI is really smart and knowledgeable and assigns a high degree of confidence/deference to its responses, the same way you might to a venerated subject matter expert or wikipedia/stackoverflow/google search result. To this person involving the AI lends more credibility/confidence in their work and their own understanding of it (vs if they just uncritically copied code off stackoverflow). Better understanding this kind of user made me realize that the quality signals and mental models people build around productivity can vary immensely even within the same profession or team. Productivity-hacking is a lot more about vibes and identity-construction/tribal affiliations than most people would like to admit.

Post reply on HN