Live data from Hacker News

Is chain-of-thought AI reasoning a mirage?

seangoedecke.com

171–180 of 191 posts

Re: Is chain-of-thought AI reasoning a mirage?

#171

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

Congratulations, you've said the quiet part out loud.

Yes, the idea is fundamentally flawed. But there's so much hype and so many dollars to be made selling such services, everyone is either genuinely fooled or sticking their fingers in their ears and pretending not to notice.

Re: Is chain-of-thought AI reasoning a mirage?

#172
post #155
post #132

Earlier quoted context omitted.

Perception and interpretation can very much be influenced by language (Sapir-Wharf hypothesis), so to the extent that perception and interpretation influence intelligence, it's not clear that the relationship is only in one direction.

Am I the exception? When thinking I don't conceptualize things in words - the compression would be too lossy. Maybe because I'm fluent in three languages (one germanic, one romance, one slavic)?

Our brains reason in many domains depending on the situation.

For domains built primarily on linguistic primitives (legal writing), we do often reason through language. In other domains (i.e spatial) we reason through vision or sound.

We experience this distinction when we study the formula vs the graph of a mathematical function, the former is linguistic, the latter is visual-spatual.

And learning multiple spoken languages is a great way to break out of particularly rigid reasoning patterns, and as important, countering biases that are influenced by your native language.

Re: Is chain-of-thought AI reasoning a mirage?

#173

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

I don't think that the concept of "real reasoning" vs simulated or fake reasoning makes any sense... LLM reasoning can be regarded as a subset of human reasoning, and a more useful comparison would be not real vs fake but rather what is missing from LLM reasoning that would need to be added (likely in a completely new architecture - not an LLM/transformer) to make it more human-like and capable.

Human reasoning, and cortical function in general, would also appear to be prediction based, but there are many differences to LLMs, starting with the fact that we learn continuously and incrementally from our own experience and prediction failures and successes. Human reasoning is basically chained what-if prediction, based on predictive outcomes of individual steps that we have learnt, either in terms of general knowledge or domain-specific problem solving steps that we have learnt.

Perhaps there is not so much difference between what a human does and an LLM does in, say, tackling a math problem when the RL-trained reasoning-LLM chains together a sequence of reasoning steps that worked before...

Where the difference come in, is in how the LLM learned those steps in the first place, and what happens when its reasoning fails. In humans these are essentially the same thing - we learn by predicting and giving it a go, and learn from prediction failure (sensory/etc feedback) to update our context-specific predictions for next time. If we reach a reasoning/predictive impasse - we've tried everything that comes to mind and everything fails, then our innate traits of curiosity and boredom (maybe more?) come to play and we will explore the problem and learn and try again. Curiosity and exploration can of course lead to gain of knowledge from things like imitation and active pursuit (or receipt) of knowledge from sources other then personal experimentation.

The LLM of course has no ability to learn (outside of in-context learning - a poor substitute), so is essentially limited in capability to what it has been pre-trained on, and pre-training is never going to be the solution to a world full of infinite ever-changing variety.

So, rather than say that an LLM isn't doing "real" reasoning, it seems more productive to acknowledge that prediction is the basis of reasoning, but that the LLM (or rather a future cognitive architecture - not a pass-thru stack of transformer layers!) needs many additional capabilities such as continual/incremental learning, innate traits such as curiosity to expose itself to learning situations, and other necessary cognitive apparatus such as working memory, cognitive iteration/looping (cf thalamo-cortical loop), etc.

Re: Is chain-of-thought AI reasoning a mirage?

#174
post #154
post #145

A regular "chatting" LLM is a document generator incrementally extending a story about a conversation between a human and a robot... And through that lens, I've been thinking "chain of thought" seems like basically the same thing but with a film noir styling-twist. The LLM is trained to include an additional layer of "unspoken" text in the document, a source of continuity which substitutes for how the LLM has no othe…

Oh wow, now I want a chain of thought rewriter that makes the combination of chat and CoT put together follow this style

Click show details in chatgpt deep research for such a tale except instead of noir it's "Office Space".

Re: Is chain-of-thought AI reasoning a mirage?

#175

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

You're not allowed to say that it's not reasoning without distinguishing what is reasoning. Absent a strict definition that the models fail and that some other reasoner passes, it is entirely philosophical.

I think it’s perfectly fine to discuss whether it’s reasoning without fully committing to any foundational theory of reasoning. There are practical things we expect from reasoning that we can operationalise.

If it’s truly reasoning, then it wouldn’t be able to deceive or to rationalize a leaked answer in a backwards fashion. Asking and answering those questions can help us understand how the research agendas for improving reasoning and improving alignment should be modified.

Re: Is chain-of-thought AI reasoning a mirage?

#176

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

> It is not a "philosophical" (by which the author probably meant "practically inconsequential") question. I didn't take it that way. I suppose it depends on whether or not you believe philosophy is legitimate

The author called it “the least interesting question possible” and contraposed it with such questions as “how accurately it reflects the actual process going on.” I don’t see any other way to take it.

Re: Is chain-of-thought AI reasoning a mirage?

#177

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

> In fact, I think in near future it will be the norm for MLLMs to "think" and "reason" without outputting a single "word". It will be outputting something, as this is the only way it can get more compute - output a token, then all context + the next token is fed through the LLM again. It might not be presented to the user, but that's a different story.

That’s the only effective way to get more compute in current production LLMs, but the field is evolving.

Re: Is chain-of-thought AI reasoning a mirage?

#178
post #109

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

You're looking at this from the perspective of what would make sense for the model to produce. Unfortunately, what really dictates the design of the models is what we can train the models with (efficiently, at scale). The output is then roughly just the reverse of the training. We don't even want AI to be an "autocomplete", but we've got tons of text, and a relatively efficient method of training on all prefixes of a…

It’s indeed a very tricky problem with no clear solution yet. But if someone finds a way to bootstrap it, it may be a new qualitative jump that may reverse the current trend of innovating ways to cut inference costs rather than improve models.

Re: Is chain-of-thought AI reasoning a mirage?

#179

How can a statistical representation of reason ever be reason itself?

Users of Eliza [1] could convince themselves that the programme was intelligent, much to the dismay of Weizenbaum. It is no surprise that people will claim that more complex (and poorly understood) applications are intelligent. If our own salesbabble and technology can be used to bamboozle us and defeat our willingness to understand, we have fully regressed to credulity and Carl Sagan's state of captured idiocy. [2]…

A tangent, but I remember having a version Eliza on our family Macintosh SE as a kid, and my brothers and I would spend way too much time typing inappropriate things trying to get a silly response.

I think you have a very important point, and I'm amazed I've never heard of the Eliza Effect.

Re: Is chain-of-thought AI reasoning a mirage?

#180

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language.

I'd claim that this assumption doesn't even hold true for humans. Reasoning in language is the most "flashy" kind of reasoning and the one that can be most readily shared with other people - because we can articulate it, write it down, publish, etc.

But I know for sure that I'm not constantly narrating my life in my head, like the reasoning traces of LLMs.

A lot of reasoning happens visually, I.e. by imagining some scene and thinking how it would play out. In other situations, it's spontaneous ideas that "just pop up" - I.e., there are unconscious processes and probably some kind of association involved.

None of that uses language.

Post reply on HN