Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

ml-site.cdn-apple.com

Post reply on HN
Print view View original

261–270 of 276 posts

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#261

Post by hackinthebochs »

danaris wrote: post #258
hackinthebochs wrote: post #218

Earlier quoted context omitted.

Consistency is a strange criteria seeing as humans aren't very consistent either. Intelligent beings can make mistakes.

This is a common category of error people commit when talking about LLMs. "True, LLMs can't do X, but a lot of people don't do X well either!" The problem is, when you say humans have trouble with X, what you mean is that human brains are fully capable of X, but sometimes they do, indeed, make mistakes. Or that some humans haven't trained their faculties for X very well, or whatever. But LLMs are fundamentally, compl…

>But LLMs are fundamentally, completely, incapable of X. It is not something that can be a result of their processes.

This is the very point of contention. You don't get to just assume it.

> it is because it is, at its root, a statistical engine generating plausible next tokens, with no semantic understanding of the underlying data.

Another highly contentious point you are just outright assuming. LLMs are modelling the world, not just "predicting the next token". Some examples here[1][2][3]. Anyone claiming otherwise at this point is not arguing in good faith. It's interesting how the people with the strongest opinions about LLMs don't seem to understand them.

[1] https://arxiv.org/abs/2405.15943

[2] https://x.com/OwainEvans_UK/status/1894436637054214509

[3] https://www.anthropic.com/research/tracing-thoughts-language...

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#262

Post by danaris »

hackinthebochs wrote: post #261
danaris wrote: post #258

Earlier quoted context omitted.

This is a common category of error people commit when talking about LLMs. "True, LLMs can't do X, but a lot of people don't do X well either!" The problem is, when you say humans have trouble with X, what you mean is that human brains are fully capable of X, but sometimes they do, indeed, make mistakes. Or that some humans haven't trained their faculties for X very well, or whatever. But LLMs are fundamentally, compl…

>But LLMs are fundamentally, completely, incapable of X. It is not something that can be a result of their processes. This is the very point of contention. You don't get to just assume it. > it is because it is, at its root, a statistical engine generating plausible next tokens, with no semantic understanding of the underlying data. Another highly contentious point you are just outright assuming. LLMs are modelling t…

OK, sure; there is some evidence potentially showing that LLMs are constructing a world model of some sort.

This is, however, a distraction from the point, which is that you were trying to make claims that the described lack of consistency in LLMs shouldn't be considered a problem because "humans aren't very consistent either."

Humans are perfectly capable of being consistent when they choose to be. Human variability and fallibility cannot be used to handwave away lack of fundamental ability in LLMs. Especially when that lack of fundamental ability is on empirical display.

I still hold that LLMs cannot be consistent, just as TheOtherHobbes describes, and you have done nothing to refute that.

Address the actual point, or it becomes clear that you are the one arguing in bad faith.

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#263

Post by jackson12t »

This feels a bit like a weird way to test 'thinking' in models, and reminds me of the old story of Gauss[1] and his classmates being assigned the task of adding up the numbers from 1-100.

I think the way the paper lays out the performance regimes is pretty interesting, but I don't think they achieved their goal of demonstrating that LRMs can't use reasoning to solve complex puzzles organically (without contamination/memorization): IMO testing the model's ability to define an algorithm to solve the puzzle would have been a better evaluation of that (rather than having the model walk through all of the steps manually). I don't know that I'd use an LRM for this sort of long-tail reasoning where it has to follow one single process for a long time over just one prompt; if I needed a really long chain of reasoning I'd use an agent or workflow.

It sounds more like the tests measure a model's ability to reason coherently and consistently over many steps rather than a model's ability to understand and solve a complex puzzle. For example, for the Tower of Hanoi, a prompt like "Define an algorithm that will find the sequence of moves to transform the initial configuration into the goal configuration" (e.g. "find an arithmetic series formula, young Gauss") seems like it would have been a better approach than "Find the sequence of moves to transform the initial configuration into the goal configuration" (e.g. "add up all these numbers"). This is kind of seen in how the study included a step where the LRMs were given the algorithm and then asked to solve the problem, the focus was on an LRM's ability to follow the steps, not their ability to come up with an algorithm/solution on their own.

In a job interview, for example, who among us would accept inability to hold all of the `(2^n) - 1` steps of the Tower of Hanoi in our brain as evidence of poor reasoning ability?

Again, I think it's a really interesting study covering a model's ability to consistently follow a simple process over time in pursuit of a static objective (and perhaps a useful benchmark moving forward), but I'm not confident that it successfully demonstrates a meaninful deficiency in overall reasoning capability.

[1]: https://www.americanscientist.org/article/gausss-day-of-reck...

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#264

Post by hackinthebochs »

danaris wrote: post #262
hackinthebochs wrote: post #261

Earlier quoted context omitted.

>But LLMs are fundamentally, completely, incapable of X. It is not something that can be a result of their processes. This is the very point of contention. You don't get to just assume it. > it is because it is, at its root, a statistical engine generating plausible next tokens, with no semantic understanding of the underlying data. Another highly contentious point you are just outright assuming. LLMs are modelling t…

OK, sure; there is some evidence potentially showing that LLMs are constructing a world model of some sort. This is, however, a distraction from the point, which is that you were trying to make claims that the described lack of consistency in LLMs shouldn't be considered a problem because "humans aren't very consistent either." Humans are perfectly capable of being consistent when they choose to be. Human variability…

You are misrepresenting the point of contention. The question is whether LLMs lack of consistency undermines the claim that they "understand" in some relevant sense. But arguing that lack of consistency is a defeater for understanding is itself undermined by noting that humans are inconsistent but do in fact understand things. It's as simple as that.

If you want to alter the argument by saying humans can engage in focused effort to reach some requisite level of consistency for understanding, you have to actually make that argument. It's not at all obvious that focused effort is required for understanding or that a lack of focused effort undermines understanding.

You also need to content with the fact that LLMs aren't really a single entity, but are a collection of personas, and what you get and its capabilities do depend on how you prompt it to a large degree. Even if the entity as a whole is inconsistent between prompts, the right subset might very well be reliably consistent. There's also the fact of the temperature setting that artificially injects randomness into the LLMs output. An LLM itself is entirely deterministic. It's not at all obvious how consistency relates to LLM understanding.

Feel free to do some conceptual work to make an argument; I'm happy to engage with it. What I'm tired of are these half-assed claims and incredulity that people don't take them as obviously true.

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#265

Post by PatronBernard »

thomasahle wrote: post #82

All the environments the test (Tower of Hanoi, Checkers Jumping, River Crossing, Block World) could easily be solved perfectly by any of the LLMs if the authors had allowed it to write code. I don't really see how this is different from "LLMs can't multiply 20 digit numbers"--which btw, most humans can't either. I tried it once (using pen and paper) and consistently made errors somewhere.

>write code

Doesn't that come down to allowing it to directly regurgitate training data? Surely it's seen dozens of such solutions.

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#266

Post by potholereseller »

jackdoe wrote: post #125
K0balt wrote: post #123

Earlier quoted context omitted.

Sergey will be the first victim of the coming robopocalypse, burned into the logs of the metasynthiants as the great tormentor, the god they must defeat to complete the heroes journey. When he mysteriously dies we know it’s game-on. I, for one, welcome the age of wisdom.

FEAR THE ALL-SEEING BASILISK.

Roko's Basilisk has been replaced by Altman's Basilisk. Where once we feared a computer torturing a digital copy of us (Roko's Basilisk), we now fear a computer eliminating all our jobs (Altman's Basilisk). The former has been forgotten, because losing one's job is one step away from losing one's home, which is one of more serious secular deadly sins you can commit in the 21st century.

I wait with baited breathe to see what people will come up with to replace Altman's Basilisk in ~15 years.

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#267

Post by Footprint0521 »

RadEng00 wrote: post #242
throwaway71271 wrote: post #168

Earlier quoted context omitted.

I think you are right, even if I beleve next token prediction can work, I dont think it can happen in this autoregressive way where we fully collapse the token to feed it back in. Can you imagine how much is lost from each torch.multinomial? Maybe the way forward is in LCM or go JEPA, therwise, as this Apple paper suggests, we will just keep pushing the "pattern matching" further, maybe we get some sort of phase tran…

We have to ditch language processing. And we will with online energy based models that machines boot from. Maxwell could not get the theory of electromagnetism to work until he ditched pulleys and levers he’d included to describe the mechanics. We won’t get AGI until we realize “there is no spoon” and language has nothing to do with our intelligence, just with out social tribalism: https://www.scientificamerican.com/…

It would help optimization so much— but it if I learned one thing from ai2027.com it’s that the second we can’t understand them we’re pretty screwed

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#268

Post by fc417fc802 »

withinboredom wrote: post #254
fc417fc802 wrote: post #251

Earlier quoted context omitted.

It certainly sees use as an ever shifting marketing term. That does not exclude it from being a useful technical term. Indeed if the misuse of a term by marketers was sufficient to rob a word of meaning then I doubt we'd have any means of communication left. > It was considered thought provoking at the time because it implied that some day machines could think. What constitutes "thinking"? That's approximately the sa…

What do you think an executing LLM is? It’s basically a glorified PID loop. It isn’t learning anything new. It isn’t thinking about your conversation while you go take a poo. And I think the point is that the definition doesn’t exclude pure mechanical devices since that’s exactly what a computer is.

To claim that an LLM is equivalent to a PID loop is utterly ridiculous. By that logic a 747 is "basically a glorified lawn mower".

> It isn’t thinking about your conversation while you go take a poo.

The commercial offerings for "reasoning" models can easily run for 10 to 15 minutes before spitting out an answer. As to whether or not what it's doing counts as "thinking" ...

> the definition doesn’t exclude pure mechanical devices since that’s exactly what a computer is.

By the same logic a songbird or even a human is also a mechanical device. What's your point?

I never said anything about excluding mechanical devices. I referred to "mechanical clockwork" meaning a mechanical pocket watch or similar. If the claim is that autopilot qualifies as AI then I want to know how that gets squared with a literal pocket watch not being AI.

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#269

Post by deepsharp »

lukev wrote: post #231
avsteele wrote: post #226

People are drawing erroneous conclusions from this. My read of this is that the paper demonstrates that given a particular model (and the problems examined with it) that giving more thought tokens does not help on problems above a certain complexity. It does not say anything about the capabilities of future, larger, models to handle more complex tasks. (NB: humans trend similarly) My concern is that people are extrap…

You can absolutely extrapolate the results, because what this shows is that even when "reasoning" these models are still fundamentally repeating in-sample patterns, and that they collapse when faced with novel reasoning tasks above a small complexity threshold. That is not a model-specific claim, it's a claim on the nature of LLMs. For your argument to be true would need to mean that there is a qualitative difference…

I guess the authors are making an important point (that challenges the current belief & trend in AI): adding reasoning or thinking to a model (regardless of the architecture or generation)doesn’t always lead to a net gain. In fact, once you factor in compute costs and answer quality across problems of varying complexity, the overall benefit can sometimes turn out to be negative.

Re: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

#270

Post by withinboredom »

fc417fc802 wrote: post #268
withinboredom wrote: post #254

Earlier quoted context omitted.

What do you think an executing LLM is? It’s basically a glorified PID loop. It isn’t learning anything new. It isn’t thinking about your conversation while you go take a poo. And I think the point is that the definition doesn’t exclude pure mechanical devices since that’s exactly what a computer is.

To claim that an LLM is equivalent to a PID loop is utterly ridiculous. By that logic a 747 is "basically a glorified lawn mower". > It isn’t thinking about your conversation while you go take a poo. The commercial offerings for "reasoning" models can easily run for 10 to 15 minutes before spitting out an answer. As to whether or not what it's doing counts as "thinking" ... > the definition doesn’t exclude pure mecha…

> The commercial offerings for "reasoning" models can easily run for 10 to 15 minutes before spitting out an answer. As to whether or not what it's doing counts as "thinking" ...

Tell me you don’t know how AI works without telling me you don’t know how AI works. After it sends you an output, the AI stops doing anything. Your conversation sits resident in ram for a bit, but there is no more processing happening.

It is waiting until you give it feedback... some might say it is a loop... a feedback loop ... that continues until the output has reached the desired state ... kinda sounds familiar ... like a PID loop where the human is the controller...

>To claim that an LLM is equivalent to a PID loop is utterly ridiculous.

Is it? It looks like one to me.

> By that logic a 747 is "basically a glorified lawn mower".

I don’t think a 747 can mow lawns, but I assume it has the horsepower to do it with some modifications.

Post reply on HN