A Knockout Blow for LLMs?
garymarcus.substack.com
A Knockout Blow for LLMs?
1–10 of 49 posts
Re: A Knockout Blow for LLMs?
#2In other news, water is wet.
I don't think anybody who uses LLMs professionally day-to-day thinks that it can reason like human beings... If some people thought this, they fundamentally do not understand how LLMs work under the hood.
Re: A Knockout Blow for LLMs?
#3It would be awesome to develop some theory around what kind of problems LLMs can and cannot solve. That should deter some leads pushing for solving the unsolvable with the technology.
That being said, this isn’t a knockout blow by any stretch. The strength of LLMs lies in the people who are excited about them. And there’s a perfect reinforcing mechanism for the excitement - the chatbots that use the models.
Admit for a second that you’re a human with biases. If you see something more frequently, you’ll think it’s more important. If you feel good when doing something, you’ll feel good about that thing. If all your friends say something, you’re likely to adopt it as your own belief.
If you have a chatbot that can talk to you more coherently than anyone you’ve ever met, and implement these two nested loops that you’ve always struggled with, you’re poised to become a fan, an enthusiast. You start to believe.
And belief is power. As in the case of neuroscience development not being able to retire the concept of the dualism of body and soul, so will the testing of LLMs not be able to retire the concept of AI poised to dominate everything soon.
Re: A Knockout Blow for LLMs?
#4The paper shows reasoning is better than no reasoning, reasoning needs more tokens to work for simple tasks, and that models get confused when things get too complicated. Nothing interesting, on the level of what an undergrad would write for a side project. If it wasn’t “from apple” no one would be mentioning it.
Re: A Knockout Blow for LLMs?
#5"They're super expensive pattern matchers that break as soon as we step outside their training distribution" - I find it really weird that things like these are seen as some groundbreaking endgame discovery about LLMs
LLMs have a real issues with polarisation. It's probably smart people saying all this stuff about knockout blows, and LLM uselessness, but I find them really useful. Is there some emperor's new clothes type thing going on here - am I just a dumbass who can't see he's excited at a random noise generator?
It's like if I saw a headline about a knockout blow for cars because SomeBigBame discovered it's possible to crash them.
It wouldn't change my normal behaviour, it would just make me think "huh, I should avoid anything SomeBigName is doing with cars then if they only just realised that."
Re: A Knockout Blow for LLMs?
#6Why would they need to execute the algorithm? That feels like complaining your fork doesn’t cut things like a knife would…
Re: A Knockout Blow for LLMs?
#7The paper shows reasoning is better than no reasoning, reasoning needs more tokens to work for simple tasks, and that models get confused when things get too complicated. Nothing interesting, on the level of what an undergrad would write for a side project. If it wasn’t “from apple” no one would be mentioning it.
I think that these kind of papers are necessary to ground people back into reality - the hype machine is too strong to be left unguarded.
Re: A Knockout Blow for LLMs?
#8The paper shows reasoning is better than no reasoning, reasoning needs more tokens to work for simple tasks, and that models get confused when things get too complicated. Nothing interesting, on the level of what an undergrad would write for a side project. If it wasn’t “from apple” no one would be mentioning it.
They cite https://arxiv.org/abs/2503.23829 which is interesting though. If you have lots of tokens to burn, just try the task lots of times. Can find better solutions than reasoning would on its first try. Only tested on small models though.
Re: A Knockout Blow for LLMs?
#9The first figure in the paper with Accuracy vs Complexity makes the whole point moot. The authors find that the performance of Claude 3.7 collapses around complexity 3 while Claude 3.7 thinking collapsed around complexity 7. A massive improvement in the complexity horizon that can be dealt with. It's real, it's quantitative, so what's the point of philosophical atguments about whether it is truly "reasoning" or not. All LLMs have various horizons, a context horizon/length, a complexity horizon etc. Reasoning pushes this out further, but not to some infinite algorithmically perfect recurrent reasoning effect. But I bet humans pretty much just have a complexity horizon of 12 or 20 or whatever and bigger models trained on bigger data with bigger reasoning posttraining and better distillation will push the horizons further and further.
Re: A Knockout Blow for LLMs?
#10> neural networks of various kinds can generalize within a training distribution of data they are exposed to, but their generalizations tend to break down outside that distribution.
so the AI companies really took that to heart and tried to put everything into the training distribution. My stuff, your stuff, their stuff. I remember the good old days of feeding the wikimedia dump into a markov chain.