Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is AI reasoning right for the wrong reasons?

quantamagazine.org

Post reply on HN
Print view View original

91–100 of 268 posts

Re: Is AI reasoning right for the wrong reasons?

#91

Post by ilovecake1984 »

vidarh wrote: post #79
Angostura wrote: post #45

Earlier quoted context omitted.

I think the article is a lot more interesting than you make out, because it isn’t really about ‘what we mean by reasoning’. It’s about do we really know what’s going on in the box - an is the ‘chain of reasoning’ indicative of what’s going on, or merely an anthropomorphised fiction that kids us into believing we understand what’s going on.

We know that human introspection is at best imperfect, and at worst outright fiction, thanks to Sperry's split brain experiments, so while it might be interesting, if the chain of reasoning isn't indicative, that would be in line with how humans work.

This is a good take. People overwhelmingly fit a narrative to how they make decisions rather than make decisions according to a chain of reasoning.

Re: Is AI reasoning right for the wrong reasons?

#92

Post by lucisferre »

I wonder if some of this explains why people have been finding with Opus 5 that running it with lower settings than "High" is producing better or at least just as good results.

> Not so fast. A 2025 paper(opens a new tab) from Northeastern University and the University of California, Berkeley on frontier open-source LRMs showed that between 30% and 60% of their “thinking steps” had “minimal causal impact” on the answers the models produced to benchmark math questions. Chop half of them out, and a model’s performance barely suffers. “We want to be careful when we review these chain-of-thought prompts because they may not be linked to the final output,” said Weiyan Shi(opens a new tab), one of the study’s authors

Re: Is AI reasoning right for the wrong reasons?

#93

Post by Glyptodon »

My limited and really non technical understanding of AI suggests that probability plays a very big role in AI behavior, and it makes sense to me that "reasoning" could be seen through a lens of creating additional content that will better constrain the probability distribution of the final output. In this case the "reasoning" done internally could be functionally bad or appear to be giberish so long as its effect on future generation is appropriate. To naive me who really doesn't know tons about AI this seems like it could be testable.

Re: Is AI reasoning right for the wrong reasons?

#94

Post by mdp2021 »

andrewla wrote: post #66
miltonlost wrote: post #51

Earlier quoted context omitted.

> I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality Ah, so you're more in the Investor mindset than the Scientist mindset. All you care about is results, not how it got there. There's a whiff of "hey, it's magic!" to that.

No, that's stupid. The question of how LLMs (and "LRM"'s, an unfortunate and by the article's own admission, already outdated term) accomplish what they accomplish is fascinating. The question of whether they reason is meaningless. Most of the article is about the latter, with tiny tiny tiny bits of the former.

> The question of whether they reason ... Most of the article is about ...

I read it as: the content of the explicit verbalized reasoning may not be that important to the outcome. I'd say that raises most important questions.

Re: Is AI reasoning right for the wrong reasons?

#95

Post by mdp2021 »

esperent wrote: post #82
janalsncm wrote: post #69

There is a long history of bad naming conventions in the field of AI, including “artificial intelligence” itself imo. (What is “intelligence” here? It’s more like “automation” or “automated problem solving”.) What really happens is that we figure out something that works, sometimes inspired by some biological thing or neuroscience thing. Examples: neural network, attention, reasoning, hallucination, agents, experts i…

Automated problem solving... of specifically the kind of problems we'd normally need human intelligence to solve (coding, translation, etc.). So "automated intelligence" would make sense to me.

> automated

Automata in context are artificial. Humans are automata, human intelligence can be said an automation. The point making a difference is that that in context is artificial.

Re: Is AI reasoning right for the wrong reasons?

#96

Post by andy99 »

ilovecake1984 wrote: post #91
vidarh wrote: post #79

Earlier quoted context omitted.

We know that human introspection is at best imperfect, and at worst outright fiction, thanks to Sperry's split brain experiments, so while it might be interesting, if the chain of reasoning isn't indicative, that would be in line with how humans work.

This is a good take. People overwhelmingly fit a narrative to how they make decisions rather than make decisions according to a chain of reasoning.

That’s just conflating reasoning and “parallel reconstruction” - there is such thing as reasoning, and I agree it’s probably less common in decision making, but it’s fundamental to many tasks where we figure out a solution, like writing an algorithm, math proofs, etc.

Re: Is AI reasoning right for the wrong reasons?

#97

Post by mdp2021 »

ForHackernews wrote: post #15
baxtr wrote: post #2

> This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning ca…

I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces. "Hey, customer service chatbot, can you help me get a refund for my order?" BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..." GOOD: "Yes, this computer system can start the refund process. Please enter your order number."

> discourage anthropomorphisation by avoiding personal pronouns

The fault is in the naïve antropomorphizer not in that which writes "I" being in fact a "first person".

Re: Is AI reasoning right for the wrong reasons?

#98

Post by ux266478 »

miltonlost wrote: post #51
andrewla wrote: post #7

I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting. Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim." I don't see a clear…

> I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality Ah, so you're more in the Investor mindset than the Scientist mindset. All you care about is results, not how it got there. There's a whiff of "hey, it's magic!" to that.

Word games aren't scientific in the slightest, and as Hume says on verbal arguments in general, they "cannot possibly be of any importance."

Re: Is AI reasoning right for the wrong reasons?

#99

Post by mdp2021 »

drob518 wrote: post #73

This feels like a problem with anthropomorphizing. We’re using words like “reasoning” and “thinking” because they are comfortable, and then we’re getting wrapped around the axle because we’re not sure if those words are totally accurate. I assure you that they aren’t accurate (the model is not alive and it’s all just a lot of matrix math under the hood), but there are no good alternative words. If we wanted to be acc…

We do not academically consider reasoning or thinking an activity restricted to humans, but as absolute processes.
Post reply on HN