Even if one-shot LLM performance has plateaued (which I'm not convinced this data shows given omission of recent models that are widely claimed to be better) that missing the point that I see in my own work. The improved tooling and agent-based approaches that I'm using now make the LLM one-shot performance only a small part of the puzzle in terms of how AI tools have accelerated the time from idea to decent code. For instance the planning dialogs I now have with Claude are an important part of what's speeding things up for me. Also, the iterative use of AI to identify, track, and take care of small coding tasks (none of which are particularly challenging in terms of benchmarks) is simply more effective. Could this all have been done with the LLM engines of late 2024. Perhaps, but I think the fine-tuning (and conceivably the system prompts) that make the current LLM's more effective at agent-centered workflows (including tool-use) are a big part of it. One-shot task performance at challenging tasks is an interesting, certainly foundational, metric. But I don't think it captures the important advances I see in how LLM's have gotten better over the last year in ways that actually matter to me. I rarely have a well-defined programming challenge and the obligation to solve it in a single-shot.
Are LLM merge rates not getting better?
31–40 of 175 posts
Re: Are LLM merge rates not getting better?
#32If you look at a separate trend for the smaller Sonnet models, you can see a rapid trend
Re: Are LLM merge rates not getting better?
#33They are getting better, but they are also hitting diminishing returns.
There's only so much data to train on, and we are unlikely to see giant leaps in performance as we did in 2023/2024.
2026-27 will be the years of primarily ecosystem/agentic improvements and reducing costs.
Re: Are LLM merge rates not getting better?
#34Earlier quoted context omitted.
This has been the general consensus for about three years now. "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain. Without any data from the study past September I think its not unreasonable, if you want to make an argument based on evidence. For me personally, I agree with you, I'm really seeing it as well.
> "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain. well, yeah. because that's been the experience for many people. 3 years ago, trying to use ChatGPT 3.5 for coding tasks was more of a gimmick than anything else, and was basically useless for helping me with my job. today, agentic Opus 4.6 provides more value to me than probably 2 more human engineers on my team would
Yep this has been my experience too.
I tried GPT3.5 for translating code from typescript to rust. It made many mistakes in rust. It couldn't fix borrow checker issues. The context was so small that I could only feed it small amounts of my program at a time. It also introduced new bugs into the algorithm.
Yesterday I had an idea for a simple macos app I wanted. I prompted claude code. It programmed the whole thing start to finish in 10 minutes, no problem. I asked it to optimize the program using a technique I came up with, and it did. I asked it to make a web version and it did. (Though for some reason, the web version needed several rounds of "it doesn't work, here's the console output").
I'm slowly coming to terms with the idea that my job is fundamentally changing. I can get way more done by prompting claude than I can by writing the code myself.
Re: Are LLM merge rates not getting better?
#35From my personal experience, they have gotten better, but they haven’t unlocked any new capabilities. They’ve just improved at what I was already using them for.
At the end of the day they still produce code that I need to manually review and fully understand before merging. Usually with a session of back-and-forth prompting or manual edits by me.
That was true 2 years ago, and it’s true now (except 2 years ago I was copy/pasting from the browser chat window and we have some nicer IDE integration now).
Re: Are LLM merge rates not getting better?
#36My experience has been that raw “one-shot intelligence” hasn’t improved as dramatically in the last year, but the workflow around the models has improved massively.
When you combine models with:
tool use
planning loops
agents that break tasks into smaller pieces
persistent context / repos
the practical capability jump is huge.
Re: Are LLM merge rates not getting better?
#37[dead]
Re: Are LLM merge rates not getting better?
#38Earlier quoted context omitted.
This has been the general consensus for about three years now. "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain. Without any data from the study past September I think its not unreasonable, if you want to make an argument based on evidence. For me personally, I agree with you, I'm really seeing it as well.
> "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain. well, yeah. because that's been the experience for many people. 3 years ago, trying to use ChatGPT 3.5 for coding tasks was more of a gimmick than anything else, and was basically useless for helping me with my job. today, agentic Opus 4.6 provides more value to me than probably 2 more human engineers on my team would
>well, yeah. because that's been the experience for many people.
Yes but this blogpost argues that at least over the course of 2024 to the end of 2025, those people were mistaken.
Re: Are LLM merge rates not getting better?
#39I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.
Re: Are LLM merge rates not getting better?
#40Yeah I'm not buying the last bit about lower MSE with one term in the model vs two (Brier with one outcome category is MSE of the probabilities). That's the sort of thing that would make me go dig to find where I fucked up the calculation.
With one term it gets more robust in the face of excluding endpoints when constructing the jackknife train/test split, I think. But you're right, it does sound fishy.
What the post is describing is just ANOVA. If removing a category improves the overall fit then fitting the two terms independently has the same optimal solution (with the two independent terms found to be identical). MSE never increases when adding a category.
This is why you have to reach to things that penalize adding parameters to models when running model comparisons.