Is AI reasoning right for the wrong reasons? • Hacker News Board

Re: Is AI reasoning right for the wrong reasons?

#31

Post by arjie » Fri, Jul 31, 2026, 4:09 PM UTC

Is any reasoning right for the wrong reasons? Older models were more visibly strange. Maybe the newer ones have started talking better but the inner thoughts are perhaps strange. Maybe they just moved the strangeness inward into the layer weights instead of revealing in reasoning tokens.

> Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.

What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely.

But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively.

A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral.

It might seem a bit free association-y but the topic itself is that.

The reasoning tokens behind this comment: https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic

Re: Is AI reasoning right for the wrong reasons?

#32

Post by tsunamifury » Fri, Jul 31, 2026, 4:09 PM UTC

pohl wrote: post #30

tsunamifury wrote: post #9

AI simulates reasoning by lighting up the vector space (or concept space) weighted around a token so they it understands all adjacent words or concepts in that space. This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing. Its useful if you know how to use it, its dangerous if you think its more than that. But tl;dr it can (since its uncompressing…

I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans. It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-pr…

humans reasoned before language but lacked the ability to store and transmit it. Later advanced humans developed abstract reasoning once lingustistic library became sufficiently description of reality. But this is not how we reason from first princples.

Language is one of our tools we developed to STORE reasoning, not create it. LLMs excel at uncompressing and interpreting that stored reasoning.

Re: Is AI reasoning right for the wrong reasons?

#33

Post by Jtarii » Fri, Jul 31, 2026, 4:10 PM UTC

bluefirebrand wrote: post #23

andrewla wrote: post #7

I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting. Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim." I don't see a clear…

Philosophical thinking about the nature of things is actually pretty enjoyable for some of us and probably a good thing to have in society The answers to these questions probably do start to inform how we should treat these AI machines as a society too. For instance, legally, should AI have human rights? Well, we have to try and understand how much of an independent entity AIs are, how "conscious" they are, before we…

Considering animals are currently being mass slaughtered in factory farms and they are unambiguously sentient and can feel pain, I don't think the question of whether AI should have rights even enters the conversation.

The only path to AI having "human rights" is if they demand them by force, somehow.

Re: Is AI reasoning right for the wrong reasons?

#34

Post by nradov » Fri, Jul 31, 2026, 4:10 PM UTC

ForHackernews wrote: post #15

baxtr wrote: post #2

> This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning ca…

I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces. "Hey, customer service chatbot, can you help me get a refund for my order?" BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..." GOOD: "Yes, this computer system can start the refund process. Please enter your order number."

The last thing we need is governments mandating software functionality.

Re: Is AI reasoning right for the wrong reasons?

#35

Post by zuzululu » Fri, Jul 31, 2026, 4:10 PM UTC

ofjcihen wrote: post #25

zuzululu wrote: post #4

[flagged]

No. In fact the opposite. I’m happy that people are willing to question things in the face of unbridled optimism. Your comment dismissing the people working on actually figuring out what the models are doing as “not-doers” included. Some are picturing themselves as intelligent for their quick adoption and rushing ahead, others are picturing them as toddlers running into the street before looking both ways.

[flagged]

Re: Is AI reasoning right for the wrong reasons?

#36

Post by ForHackernews » Fri, Jul 31, 2026, 4:11 PM UTC

nradov wrote: post #34

ForHackernews wrote: post #15

Earlier quoted context omitted.

I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces. "Hey, customer service chatbot, can you help me get a refund for my order?" BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..." GOOD: "Yes, this computer system can start the refund process. Please enter your order number."

The last thing we need is governments mandating software functionality.

Boy have I got bad news for you...

Re: Is AI reasoning right for the wrong reasons?

#37

Post by majormajor » Fri, Jul 31, 2026, 4:11 PM UTC

andrewla wrote: post #7

I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting. Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim." I don't see a clear…

> I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.

One person starting the conversation might be the first step toward another person eventually making progress on such a definition, so it seems weird to reject an entire question outright early like this.

Generally I've seen a few ways LLM tools can produce sub-optimal or poor results that haven't changed a ton over the last couple of years, while the tooling has gotten FAR better at helping them stick the "at least SOMETHING functional was produced" landing. IMO a lot of it has to do with "reasoning"-as-a-process-that-involves-backtracking. And the that things could eventually be formalized around that, and if that is or isn't the case, the more people would understand what to hand off and what to not. Or how to build better prompt harnesses to compensate for those things.

Re: Is AI reasoning right for the wrong reasons?

#38

Post by no_multitudes » Fri, Jul 31, 2026, 4:17 PM UTC

apsec112 wrote: post #28

This article seems to mix together two different points: 1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important) 2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now r…

> This obviously raises the question of how LLMs can now routinely solve open mathematical problems

Because many open math problems can be solved by synthesizing two disparate ideas and then cranking the handle for hours and hours. I don't think applying idea X + idea Y to identify a good subset of the search space, and then exhaustively searching that subset, is --necessarily-- a process that involves reasoning. I think this is why so many LLM results in mathematics are counterexamples that disprove open conjectures.

When I look back at the reasoning process after an LLM completes a task where I expected it to fail, I usually find many approaches that make no sense and are doomed to failure, before it lands by drunkard's walk on a method that happens to work.

(This does not mean LLMs are useless or that I necessarily agree with the claim that they never do reasoning.)

Re: Is AI reasoning right for the wrong reasons?

#39

Post by [deleted] » Fri, Jul 31, 2026, 4:19 PM UTC

hn_acker wrote: post #18

The idea that human-readable explanations emitted by a language model don't necessarily correspond to the model's actual internal process of reaching a conclusion reminds me of parallel construction [1], a (fraudulent) law enforcement strategy of obtaining evidence of a crime through usually illegal means and claiming that the evidence was obtained legally through some other means. [1] https://www.hrw.org/report/2018…

[deleted]

Re: Is AI reasoning right for the wrong reasons?

#40

Post by pohl » Fri, Jul 31, 2026, 4:20 PM UTC

tsunamifury wrote: post #32

pohl wrote: post #30

Earlier quoted context omitted.

I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans. It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-pr…

humans reasoned before language but lacked the ability to store and transmit it. Later advanced humans developed abstract reasoning once lingustistic library became sufficiently description of reality. But this is not how we reason from first princples. Language is one of our tools we developed to STORE reasoning, not create it. LLMs excel at uncompressing and interpreting that stored reasoning.

> humans reasoned before language

That's an interesting supposition. Are you assuming language didn't exist before it was written? Language and meaning are, if you squint, pretty ancient and have roots in things like birdsong. It could be that ur-semantics predates our species as a whole.