Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illusion of Thinking: A Reality Check on AI Reasoning

leotsem.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#11

Post by saithound »

vincnetas wrote: post #9
salviati wrote: post #3

Earlier quoted context omitted.

If you ask me to solve increasingly dififcult Tower of Hanoi problems, I don't expect to be good at it. Neither would I expect a fellow human to be. So based on this should we question our intelligence? I heard about that paper through an "AI explained" video [0], so I might be biased, but I agree with that video that the Apple paper is "meh" at best: it points out LLM limitations that are hardly a surprise. [0] http…

Probably the difference between you and AI is that you would acknowledge that it's too difficult for you, and not to bullshit your way through.

That's _exactly_ what the LLM did: the article's authors decided to count that as a failure.

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#12

Post by pyman »

antirez wrote: post #6

The chain of thoughts is not where the reasoning capabilities of a model happens: models have reasoning capabilities that are part of the next token inference, what CoT does is searching/sampling the model space of representations and notions in order to "ground" the final reply, putting in the context window in an explicit way all the related knowledge and ideas the model possess about the question. It is absolutely…

What they're saying is that pattern-matching isn't the path to AGI. Humans and AI can both solve the Tower of Hanoi, but once the number of disks goes up, we both struggle.

Apple's point is that if we want to build something smarter than us, we need to look at intelligence and reasoning from a different angle.

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#13

Post by vincnetas »

saithound wrote: post #11
vincnetas wrote: post #9

Earlier quoted context omitted.

Probably the difference between you and AI is that you would acknowledge that it's too difficult for you, and not to bullshit your way through.

That's _exactly_ what the LLM did: the article's authors decided to count that as a failure.

Hm was reading only TFA not the research paper. But TFA mentions this :

  Perhaps the most unsettling finding is what failure looks like. Even when models are completely wrong, they sound persuasive. The reasoning is fluent, the explanations are structured, and the conclusions are confidently delivered. But the logic doesn’t hold.

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#14

Post by pyman »

crowie wrote: post #10

This might be a dumb question, and will inevitably showcase my ignorance in this field to others, but I will risk that; Why can't AI at a certain level execute algorithms with solutions that have been proved to work for a very long time? What I mean is, the solution of the Hanoi towers problem is known. It does not take a lot of computational power to achieve the result. What is stopping an AI such as the objects of…

If humanity moves to Mars one day and leaves behind all the AI servers running on solar power, then comes back a billion years later, the AI would still be saying the same things. Why? Because no matter how powerful it is, AI doesn't evolve or grow on its own.

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#15

Post by crowie »

pyman wrote: post #14
crowie wrote: post #10

This might be a dumb question, and will inevitably showcase my ignorance in this field to others, but I will risk that; Why can't AI at a certain level execute algorithms with solutions that have been proved to work for a very long time? What I mean is, the solution of the Hanoi towers problem is known. It does not take a lot of computational power to achieve the result. What is stopping an AI such as the objects of…

If humanity moves to Mars one day and leaves behind all the AI servers running on solar power, then comes back a billion years later, the AI would still be saying the same things. Why? Because no matter how powerful it is, AI doesn't evolve or grow on its own.

Gotcha, but I didn't mean it in that way. What I meant is, that problems like the case-study ones don't need a revolutionary nor an original answer which would require growth, they can be solved with old solutions which I would assume would be in some way embedded into the learning dataset of these models. Yeah, the scope of the problem is bigger, but the correct answer should come down in any case to a correct implementation of the known algorithm. The thing I'm asking is what causes the hindrance which prevents these AIs from performing in appropriate ways given old problems and old solutions.

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#16

Post by rcarmo »

vincnetas wrote: post #13
saithound wrote: post #11

Earlier quoted context omitted.

That's _exactly_ what the LLM did: the article's authors decided to count that as a failure.

Hm was reading only TFA not the research paper. But TFA mentions this : Perhaps the most unsettling finding is what failure looks like. Even when models are completely wrong, they sound persuasive. The reasoning is fluent, the explanations are structured, and the conclusions are confidently delivered. But the logic doesn’t hold.

That sounds a lot like a salesperson. And yes, there is a human tendency to twist reasoning to make the written word look polished, and I don’t think LLM training has fixed that bias.

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#17

Post by rcarmo »

antirez wrote: post #6

The chain of thoughts is not where the reasoning capabilities of a model happens: models have reasoning capabilities that are part of the next token inference, what CoT does is searching/sampling the model space of representations and notions in order to "ground" the final reply, putting in the context window in an explicit way all the related knowledge and ideas the model possess about the question. It is absolutely…

Exploring how to consistently arrive at a negative result is still a valid research goal. I don’t think we’ve had enough of that kind of research regarding LLMs—-everything is so positive that it defies basic statistics…

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#18

Post by ryandvm »

pyman wrote: post #14
crowie wrote: post #10

This might be a dumb question, and will inevitably showcase my ignorance in this field to others, but I will risk that; Why can't AI at a certain level execute algorithms with solutions that have been proved to work for a very long time? What I mean is, the solution of the Hanoi towers problem is known. It does not take a lot of computational power to achieve the result. What is stopping an AI such as the objects of…

If humanity moves to Mars one day and leaves behind all the AI servers running on solar power, then comes back a billion years later, the AI would still be saying the same things. Why? Because no matter how powerful it is, AI doesn't evolve or grow on its own.

I like your thought experiment and I think you're correct, but that's because we never gave it the physical possibility of a feedback loop (a.k.a. evolution).

I think if you added a step where the LLMs tweak their own build process and redeploy, your experiment would have wildly different results.

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#19

Post by Yizahi »

The so called "reasoning" of LLM programs is really a sham. And authors of those programs are sometimes expose it themselves. For example the article by Anthropic about Claude "reasoning". When they get to the math block they ask the program to add two numbers and then ask to write step by step flow how the LLM did it. LLM generates a human-based flow, because that's what it copied from the training data, while the real flow of LLM adds numbers is vastly different.

Basically so called "reasoning" is just generation of additional intermediary output, resembling real reasoning, but not being it.

https://transformer-circuits.pub/2025/attribution-graphs/bio...

Re: The Illusion of Thinking: A Reality Check on AI Reasoning

#20

Post by rsynnott »

> Apple’s new paper, The Illusion of Thinking, quietly released ahead of WWDC 2025, challenges many of the assumptions we’ve come to rely on in the LLM space.

So... wait, were people _really_ assuming that these things were reasoning? Why? Like, because the marketing said so? I had the idea that that was generally viewed as puffery; obviously they're not reasoning.

It's an interesting paper, but its outcome is completely unsurprising. What would have been surprising is if it had shown something different.

> Perhaps the most unsettling finding is what failure looks like. Even when models are completely wrong, they sound persuasive.

Again... This has been a fairly well-known problem with LLMs since GPT-3 or so. I'm not sure why anyone would find it unsettling at this point; they're confident-sounding bullshit engines.

Post reply on HN