never been a better time to be a swe who doesnt or significantly limits the use of AI agents
We are changing our developer productivity experiment design
11–20 of 62 posts
Re: We are changing our developer productivity experiment design
#12It's kind of funny that METR is known primarily for both the most bearish study on AI progress (the original 20% slowdown one), and the most bullish one on AI progress (the long-task horizon study showing exponential increase in duration of tasks AI models can accomplish with respect to date of release).
In either case, it seems people ended up bolstering their preexisting views on AI based on whichever study most affirmed them (for the former, that AI coding models didn't actually help and created a mirage of productivity that required more work to fix than was worth it, the latter that AI models were improving at an exponential rate and will invariably eclipse SWE's in all tasks in a deterministic amount of time.)
I think the truth is somewhere in the middle. Just anecdotally we've seen multi-million dollar fortunes being minted by small teams developing using 90% AI-assisted coding. Anthropic claims they solely use agents to code and don't modify any code manually.
Re: We are changing our developer productivity experiment design
#13Those developer quotes are tough to read. Rate limits are going to hit like a truck when the labs eventually need to make a profit.
At this point the AI labs would pretty much have to form an illegal price fixing cartel in order to jack the prices up, they've been competing to drive down prices for so long.
They'd have to get the Chinese AI labs to go along with that price fixing too.
Re: We are changing our developer productivity experiment design
#14This is very interesting because I see a lot of AI detractors point to the original study as proof that AI is overhyped and nothing to worry about. In this new study the findings are essentially reversed (20% slowdown to 20% speedup).
AI detractors loved that previous study so much. It seems to have been brought up in the majority of conversations about AI productivity over the past six months.
(Notable to me was how few other studies they cited, which I think is because studies showing AI productivity loss are quite uncommon.)
Re: We are changing our developer productivity experiment design
#15"I don't want to do this without AI" sounds like we're already well into the brain atrophy stage of this. Now what? (I'd think about it myself but....)
Re: We are changing our developer productivity experiment design
#16Those developer quotes are tough to read. Rate limits are going to hit like a truck when the labs eventually need to make a profit.
At this point the AI labs would pretty much have to form an illegal price fixing cartel in order to jack the prices up, they've been competing to drive down prices for so long. They'd have to get the Chinese AI labs to go along with that price fixing too.
They’d have an entire country of geniuses prepared to defend against the antitrust allegations, who’s to stop them? /s
Re: We are changing our developer productivity experiment design
#17Really interesting updates to their 2025 experiment. Repeat devs from the original experiment went from 0-40% slowdown to now -10-40% speedup - and METR estimates this as a 'lower-bound' more devs saying they dont even want to do 50% of their work without AI, even for 50/hr 30-50% of devs decided not to submit certain tasks without AI, missing the tasks with the highest uplift it also seems like there is a skill gap…
The finding of the first study was people cannot judge their performance with these tools. So I don’t think the lack of individuals not willing to work without them is indicative of productivity improvements. I think it’s indicative of them being enjoyable to use.
Re: We are changing our developer productivity experiment design
#18"I don't want to do this without AI" sounds like we're already well into the brain atrophy stage of this. Now what? (I'd think about it myself but....)
"I avoid issues like AI can finish things in just 2 hours, but I have to spend 20 hours. I will feel so painful if the task is decided as AI-disallowed."
What really doesn't sound like the results they got where developers may get up to twice as productive on the best scenario.
There's surely something scary there. And the lack of people ambivalent about AI isn't a certain indication it's well accepted as they think, it can just as easily be caused by polarization.
Re: We are changing our developer productivity experiment design
#19"I don't want to do this without AI" sounds like we're already well into the brain atrophy stage of this. Now what? (I'd think about it myself but....)
AI will soon be an intrinsic part of the job. Now what? "Get your thumb out of your ass and learn [how to use AI]." —Eric S. Raymond
Re: We are changing our developer productivity experiment design
#20It's kind of funny that METR is known primarily for both the most bearish study on AI progress (the original 20% slowdown one), and the most bullish one on AI progress (the long-task horizon study showing exponential increase in duration of tasks AI models can accomplish with respect to date of release). In either case, it seems people ended up bolstering their preexisting views on AI based on whichever study most af…
> Anthropic claims they solely use agents to code and don't modify any code manually.
Have you used CC? It shows. They did not make their fortune off this, and it’s at least lost me a customer because of how sloppy it is. The model is good, and it’s why they have to gate access to it. I’d much rather use a different harness.
I do think you’re on to something though. As societal wealth further concentrates among the few, we’re going to get more and more slop for the rest of us because we have no money (relatively speaking). Agentic coding is here to stay because we as a society are forced more and more slop. It’s already rampant, this is just automating it.