Earlier quoted context omitted.
Performance review at FAANGs has always been vibes and soft skills.
Shipped projects don't lie.
Come on, we all know that shipping a project doesn't mean the project was productive
The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
111–115 of 115 posts
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#112Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#113Earlier quoted context omitted.
> 'good' models have been available since the start of the year today: https://www.anthropic.com/news/redeploying-fable-5 35 days ago: https://www.anthropic.com/news/claude-opus-4-8 70 days ago: https://openai.com/index/introducing-gpt-5-5/ 77 days ago: https://www.anthropic.com/news/claude-opus-4-7 119 days ago: https://openai.com/index/introducing-gpt-5-4/ 182 days ago: The start of the year
Opus 4.5/4.6 are what many people consider the first 'good' models and it's from last year/start of this year. But fine, let's say everything before gpt 5.5 was unusable crap. Then there should still be projects that would normally have previously taken ~2 years done in just two months. Where are they?
I’m starting to see huge solo projects with 100 commits per day turning up on GitHub.
Last year these were unusable slop.
Now they’re “getting there”. Not quite as good as hand crafted code written by humans, but usable.
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#114Earlier quoted context omitted.
> Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. I built a UAT agent on top of claude-agent-sdk, it uses Playwright and can spin up a preview instance for PRs we open. It uses its knowledge of the code to create a test plan, runs that test plan, and takes screenshots as it does so. On a recent PR I made a change to our MFA implementation and assumed that I would nee…
I built a very similar tool recently mentioned elsewhere in these comments. I think with the current state of LLMs, harnesses, and related tooling, being able to create or setup self-eval tooling is the biggest differentiator between merely using LLMs to write code vs realizing true 10x productivity wins. I'm curious whether this is something LLMs are eventually going to be good enough at doing, or something the aver…
Ah, it's an internal tool but maybe worth open-sourcing at some point! :)
re: price, definitely on the pricier side relative to our average UAT run but this one was very comprehensive due to the increased risk of issues when changing MFA... and relative to a human's time, even a cheaper QA person. I would estimate ~90 minutes of human time to figure out the needed flows and then go through the same steps on this specific change (potentially getting stuck on several pieces which the agent was able to avoid via code inspection).
Re: The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
#1152025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/…
Models might be better today, but the takeaway that persists is the delusion around things being magically faster.