Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLM merge rates not getting better?

entropicthoughts.com

Post reply on HN
Print view View original

71–80 of 175 posts

Re: Are LLM merge rates not getting better?

#71

Post by BoppreH »

Controversial opinion from a casual user, but state-of-art LLMs now feel to me more intelligent then the average person on the steet. Also explains why training on more average-quality data (if there's any left) is not making improvements.

But LLMs are hamstrung by their harnesses. They are doing the equivalent of providing technical support via phone call: little to no context, and limited to a bidirectional stream of words (tokens). The best agent harnesses have the equivalent of vision-impairment accessibility interfaces, and even those are still subpar.

Heck, giving LLMs time to think was once a groundbreaking idea. Yesterday I saw Claude Code editing a file using shell redirects! It's barbaric.

I expect future improvements to come from harness improvements, especially around sub agents/context rollbacks (to work around the non-linear cost of context) and LLM-aligned "accessibility tools". That, or more synthetic training data.

Re: Are LLM merge rates not getting better?

#72

Post by jygg4 »

cj wrote: post #21
jeffnv wrote: post #9

I don't think it's true, but am I alone in wishing it was? My world is disrupted somewhat but so far I don't think we have a thing that upends our way of life completely yet. If it stayed exactly this good I'd be pretty content.

I agree with your sentiment, but I think we've yet to see the full application of the current technology. (Even if LLMs themselves don't improve, there's significant opportunity for people to use it in ways not currently being done)

The issue with llm’s is trust.

I don’t see that ever going away. Humans have learned to trust other humans over a large time scale with rules in place to control behaviour.

Re: Are LLM merge rates not getting better?

#73

Post by NewLogic »

AussieWog93 wrote: post #43
sunaurus wrote: post #39

I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.

I'd agree with you on 4.5 to 4.6, but going from gpt-5 or 4.0 to 4.5 was night and day.

Because post 4.0 dropped the sycophancy?

Re: Are LLM merge rates not getting better?

#74

Post by jwpapi »

thomascgalvin wrote: post #27

Anecdotally, I haven't seen any real improvement from the AI tools I leverage. They're all good-ish at what they do, but all still lie occasionally, and all need babysitting. I also wonder how much of the jump in early 2025 comes from cultural acceptance by devs, rather than an improvement in the tools themselves.

It’s better pre and post training + better harnessing

Re: Are LLM merge rates not getting better?

#75

Post by jygg4 »

aerhardt wrote: post #61

I feel that two things are true at the same time: 1) Something happened during 2025 that made the models (or crucially, the wrapping terminal-based apps like Claude Code or Codex) much better. I only type in the terminal anymore. 2) The quality of the code is still quite often terrible. Quadruple-nested control flow abounds. Software architecture in rather small scopes is unsound. People say AI is “good at front end”…

The models lose the ability to inject subtle and nuance stuff as they scale up, is what I’ve observed.

Re: Are LLM merge rates not getting better?

#76

Post by delichon »

Yesterday I asked a frontier model to help generate a report. It said great, it can do that, and output a table. I asked it to evaluate its prompt compliance in the result. It concluded that it had failed on every requirement. I asked why it had expressed such confidence, was it analagous to narcissism or psycopathy? It said no, and then said that if I just had to anthropomorphize it, I should think of it as a brilliant friend with severe frontal lobe brain damage.

That actually helps.

Re: Are LLM merge rates not getting better?

#77

Post by xyzsparetimexyz »

BoppreH wrote: post #71

Controversial opinion from a casual user, but state-of-art LLMs now feel to me more intelligent then the average person on the steet. Also explains why training on more average-quality data (if there's any left) is not making improvements. But LLMs are hamstrung by their harnesses. They are doing the equivalent of providing technical support via phone call: little to no context, and limited to a bidirectional stream…

Steet? Do you mean street? They're smarter in the same way a search engine is smarter.

Re: Are LLM merge rates not getting better?

#78

Post by jygg4 »

orwin wrote: post #65
aerhardt wrote: post #61

I feel that two things are true at the same time: 1) Something happened during 2025 that made the models (or crucially, the wrapping terminal-based apps like Claude Code or Codex) much better. I only type in the terminal anymore. 2) The quality of the code is still quite often terrible. Quadruple-nested control flow abounds. Software architecture in rather small scopes is unsound. People say AI is “good at front end”…

> People say AI is “good at front end” I only say that because I'm a shit frontend dev. Honestly, I'm not that bad anymore, but I'm still shit, and the AI will probably generate better code than I will.

As long as humans are needed to review code, it sounds your role evolves toward prompting and reviewing.

Which is akin to driving a car - the motor vehicle itself doesn’t know where to go. It requires you to prompt via steering and braking etc, and then to review what is happening in response.

That’s not necessarily a bad thing - reviewing code ultimately matters most. As long as what is produced is more often than not correct and legible.. now this is a different issue for which there isn’t a consensus across software engineer’s.

Re: Are LLM merge rates not getting better?

#79

Post by mike_hearn »

dwedge wrote: post #11
mike_hearn wrote: post #3

That's an interesting claim, but I don't see it in my own work. They have got better but it's very hard to quantify. I just find myself editing their work much less these days (currently using GPT 5.4).

Without meaning to sound dismissive, because I'm really not intending to, there's also the possibility that you've gotten worse after enough time using them. You're treating yourself as a constant in this, but man cannot walk in the same river twice.

That's a possibility, but I doubt it. I've been programming for 35 years and know what I like in code. I've also previously maintained a long review prompt in which I tell the models all the ways in which they get things wrong and to go look for/fix those problems. But those review passes now don't take as long because there are fewer such problems to begin with.

In particular GPT 5.4 is much better at not duplicating code unnecessarily. It'll take the time to refactor, to search for pre-existing utility functions, etc.

Re: Are LLM merge rates not getting better?

#80

Post by kqr »

fluidcruft wrote: post #40
kqr wrote: post #28

Earlier quoted context omitted.

With one term it gets more robust in the face of excluding endpoints when constructing the jackknife train/test split, I think. But you're right, it does sound fishy.

What the post is describing is just ANOVA. If removing a category improves the overall fit then fitting the two terms independently has the same optimal solution (with the two independent terms found to be identical). MSE never increases when adding a category. This is why you have to reach to things that penalize adding parameters to models when running model comparisons.

No, the post is doing cross-validation to test predictive power directly. The error will not decompose as neatly then.
Post reply on HN