Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLM merge rates not getting better?

entropicthoughts.com

Post reply on HN
Print view View original

41–50 of 175 posts

Re: Are LLM merge rates not getting better?

#41

Post by utopiah »

egwor wrote: post #30
thomascgalvin wrote: post #27

Anecdotally, I haven't seen any real improvement from the AI tools I leverage. They're all good-ish at what they do, but all still lie occasionally, and all need babysitting. I also wonder how much of the jump in early 2025 comes from cultural acceptance by devs, rather than an improvement in the tools themselves.

I think it depends on what you're using it for. If it is a simple kubernetes config then the model doesn't matter too much. Contract that with writing the scenario for a backtest for an algo that trades on a venue: it is not the same complexity and the basic models are terrible. I've had it tell me that it has added tests to find that they're just stubs! Opus seems to be getting there, but on more complex tasks the o…

> If it is a simple kubernetes config then the model doesn't matter too much

I guess at least this person https://www.tomshardware.com/tech-industry/artificial-intell... might disagree. I think already to know what Kubernetes even is requires quite a bit of knowledge. Using a tool that manipulate its configuration files IN PRODUCTION without risking data loss is another ball game entirely.

Re: Are LLM merge rates not getting better?

#43

Post by AussieWog93 »

sunaurus wrote: post #39

I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.

I'd agree with you on 4.5 to 4.6, but going from gpt-5 or 4.0 to 4.5 was night and day.

Re: Are LLM merge rates not getting better?

#45

Post by eterm »

roxolotl wrote: post #12

These studies are always really hard to judge the efficacy of. I would say though the most surprising thing to me about LLMs in the past year is how many people got hyped about the Opus 4.5 release. Having used Claude Code at work since it was released I haven't really noticed any step changes in improvement. Maybe that's because I've never tried to use it to one shot things? Regardless I'm more inclined to believe t…

I used it with Sonnet 4.0 a lot, and there was vastly more back-and-forth and correction of "dumb" things, such as forgetting to add "using" statements in C# files.

I don't know if it's model, or harness improvements, or inbuilt-memory or all of the above, but it often has a step where it'll check itself that is done now before trying to build and getting an inevitable failure.

Those small things add up to a much smoother and richer experience today compared to 6 months ago.

Re: Are LLM merge rates not getting better?

#46

Post by pu_pe »

Benchmaxxing aside, if you are using those tools for programming on a regular basis it should be self-evident that they are improving. I find it very hard to believe that someone using LLMs today vs what was available one year ago (Claude Code released Feb 2025) would have any difficulty answering this question.

Re: Are LLM merge rates not getting better?

#47

Post by wongarsu »

I don't find this very compelling. If you look at the actual graph they are referencing but never showing [1] there is a clear improvement from Sonnet 3.7 -> Opus 4.0 -> Sonnet 4.5. This is just hidden in their graph because they are only looking at the number of PRs that are mergable with no human feedback whatsoever (a high standard even for humans).

And even if we were to agree that that's a reasonable standard, GPT 5 shouldn't be included. There is only one datapoint for all OpenAI models. That data point more indicative of the performance of OpenAI models (and the harness used) than of any progression. Once you exclude it it matches what you would expect from a logistic model. Improvements have slowed down, but not stopped

1: https://metr.org/assets/images/many-swe-bench-passing-prs-wo...

Re: Are LLM merge rates not getting better?

#49

Post by rustyhancock »

thomascgalvin wrote: post #27

Anecdotally, I haven't seen any real improvement from the AI tools I leverage. They're all good-ish at what they do, but all still lie occasionally, and all need babysitting. I also wonder how much of the jump in early 2025 comes from cultural acceptance by devs, rather than an improvement in the tools themselves.

I think I'm coming to the same conclusion Gpt-3 to 5.3 have had real tangible but incremental improvements with quite diminishing returns.

Perhaps we won't see a phase change like improvement as we did from gpt-2 through to 3 until there is several more orders of magnitude parameters and/or training. Perhaps we will never see it again!

What is getting rapidly better is scaffolding but this seems to be more about understanding and building tools around LLMs than the LLMs themselves improving.

I'm still excited about AI but not constantly hyped to the rafters as some.

Re: Are LLM merge rates not getting better?

#50

Post by pnathan »

Data is missing on this chart.

It's my experience that opus 4, and then, particularly, 4.5, in Claude code, are head and shoulders above the competition.

I wrote an agentic coder years ago and it yielded trash. (Tried to make it do then what kiro does today).

The models are better. Now, caveat - I don't use anything but opus for coding - Sonnet doesn't do the trick. My experience with Codex and Gemini is that their top models are as good as Sonnet for coding...

Post reply on HN