Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Order Doesn’t Matter, But Reasoning Does

arxiv.org

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Order Doesn’t Matter, But Reasoning Does

#11

Post by philipov »

spaintech wrote: post #2

When a language model is trained for chain-of-thought reasoning, particularly on datasets with a limited number of sequence variations, it may end up memorizing predetermined step patterns that seem effective but don’t reflect true logical understanding. Rather than deriving each step logically from the previous ones and the given premises, the model might simply follow a “recipe” it learned from the training data. A…

> As a result, this adherence to learned patterns can overshadow genuine logical relationships, causing the model to rely on familiar sequences instead of understanding why one step logically follows from another.

To be honest, even humans rarely get above this level of understanding for many tasks. I don't think most people really understand math above the level of following the recipes they learned by rote in school.

Or beyond following the runbook in their IT department's documentation system.

And when the recipe doesn't work, they are helpless to figure out why.

Re: Order Doesn’t Matter, But Reasoning Does

#13

Post by [deleted] »

kingkongjaffa wrote: post #7
spaintech wrote: post #2

When a language model is trained for chain-of-thought reasoning, particularly on datasets with a limited number of sequence variations, it may end up memorizing predetermined step patterns that seem effective but don’t reflect true logical understanding. Rather than deriving each step logically from the previous ones and the given premises, the model might simply follow a “recipe” it learned from the training data. A…

> instead of understanding why one step logically follows from another There’s currently 0% chance of “understanding” happening at any point with this technology.

[deleted]

Re: Order Doesn’t Matter, But Reasoning Does

#14

Post by ikanreed »

kingkongjaffa wrote: post #7
spaintech wrote: post #2

When a language model is trained for chain-of-thought reasoning, particularly on datasets with a limited number of sequence variations, it may end up memorizing predetermined step patterns that seem effective but don’t reflect true logical understanding. Rather than deriving each step logically from the previous ones and the given premises, the model might simply follow a “recipe” it learned from the training data. A…

> instead of understanding why one step logically follows from another There’s currently 0% chance of “understanding” happening at any point with this technology.

I mostly agree, but struggle with saying this with perfect certainty.

Understanding in the "have mental model of the world, apply it, derive thoughts from that model, derive words from thoughts" pattern is a thing they don't do the way we do.

But understanding of some kinds CAN be encoded into tokens and their relationships. They're clearly capable of novel, correct inferences, that are not directly contained within their training sets.

I all-but-guarantee my "My fish suffocated when I brought it to space, even though I gave it a space suit filled with pure water, why?" test case is not something it was explicitly trained on, but it correctly inferred "Because fish need oxygenated water"

Re: Order Doesn’t Matter, But Reasoning Does

#15

Post by scribu »

The HN submission title is editorialized in a non-helpful way. Why beat a dead horse instead of focusing on what’s actually new in TFA?

The linked paper proposes an obvious-in-retrospect form of data augmentation: shuffle the order of the premises, so that the model can’t rely on spurious patterns. That’s kinda neat.

Re: Order Doesn’t Matter, But Reasoning Does

#16

Post by joe_the_user »

spaintech wrote: post #2

When a language model is trained for chain-of-thought reasoning, particularly on datasets with a limited number of sequence variations, it may end up memorizing predetermined step patterns that seem effective but don’t reflect true logical understanding. Rather than deriving each step logically from the previous ones and the given premises, the model might simply follow a “recipe” it learned from the training data. A…

That claim sounds like a quote from a paper but it's not from the currently linked paper. The paper itself seems more like antidote to the problem and does seem to roughly assume the claim.

I like the claim and I'd guess it's true but this seems like a weird way to introduce it.

Re: Order Doesn’t Matter, But Reasoning Does

#17

Post by TeMPOraL »

jvanderbot wrote: post #9
kingkongjaffa wrote: post #7

Earlier quoted context omitted.

> instead of understanding why one step logically follows from another There’s currently 0% chance of “understanding” happening at any point with this technology.

How do we define understanding?

There are many ways to define it. Taking, for example the "definition" from Wikipedia here[0], you could say that LLMs are understanding, in a distilled form, because relationships is precisely what they're made of.

--

[0] - https://en.wikipedia.org/wiki/Understanding#Definition - though this feels more like vague musings than a definition proposal.

Re: Order Doesn’t Matter, But Reasoning Does

#18

Post by spaintech »

scribu wrote: post #15

The HN submission title is editorialized in a non-helpful way. Why beat a dead horse instead of focusing on what’s actually new in TFA? The linked paper proposes an obvious-in-retrospect form of data augmentation: shuffle the order of the premises, so that the model can’t rely on spurious patterns. That’s kinda neat.

Correct, I updated the title of the original paper. Thank you your bringing up.

Re: Order Doesn’t Matter, But Reasoning Does

#19

Post by spaintech »

If an LLM’s logic is derived primarily from its training phase… essentially, by following patterns it has previously seen; doesn’t that underscore the critical role of training? We invest significantly in reinforcement learning and subsequent processes, so if the paper’s claim is accurate, perhaps we need to explore innovative approaches during the training phase
Post reply on HN