Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Procedural knowledge in pretraining drives reasoning in large language models

arxiv.org

Post reply on HN
Print view View original

11–20 of 104 posts

Re: Procedural knowledge in pretraining drives reasoning in large language models

#11

Post by spitfire »

spitfire wrote: post #6
dinfinity wrote: post #4

Earlier quoted context omitted.

No. They're saying that the model they analyzed used mainly information on _how_ to solve math problems from its training data, rather than documents that contained the answers to the (identical) math problems: > "We investigate which data influence the model’s produced reasoning traces and how those data relate to the specific problems being addressed. Are models simply ‘retrieving’ answers from previously seen pret…

What I further got from this is the models are learning the methods, but not evaluating themselves along the way. They don’t check for errors. So once they go down a path they can’t properly backtrack. This feels like the ground truth I’ve experienced in LLMs to date.

I’ll add when I say “learning” I mean memorization. Memorizing on a higher level than facts.

I would love to spend the time and see how altering the query alters the reasoning path. How firm is in the path once it’s chosen?

A high level approach has the possibility to be very computer efficient.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#12

Post by jebarker »

jpcom wrote: post #9
mattdeboard wrote: post #8

Earlier quoted context omitted.

No. If I'm understanding correctly it means the software is learning how to solve problems in general by ingesting examples of procedural problem-solving.

You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem. The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how…

> In essence, it's like a neural net learning to follow a recipe by watching a chef cook—rather than inventing its own recipes entirely from first principles.

Just like how a chef learns

Re: Procedural knowledge in pretraining drives reasoning in large language models

#13

Post by Retric »

jebarker wrote: post #12
jpcom wrote: post #9

Earlier quoted context omitted.

You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem. The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how…

> In essence, it's like a neural net learning to follow a recipe by watching a chef cook—rather than inventing its own recipes entirely from first principles. Just like how a chef learns

A chef also learns through trial and error not just reading how others have cooked in the past and then copping their motions.

This is exemplified by how altitude has a meaningful impact but isn’t discussed for a given recipe.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#14

Post by NitpickLawyer »

spitfire wrote: post #6
dinfinity wrote: post #4

Earlier quoted context omitted.

No. They're saying that the model they analyzed used mainly information on _how_ to solve math problems from its training data, rather than documents that contained the answers to the (identical) math problems: > "We investigate which data influence the model’s produced reasoning traces and how those data relate to the specific problems being addressed. Are models simply ‘retrieving’ answers from previously seen pret…

What I further got from this is the models are learning the methods, but not evaluating themselves along the way. They don’t check for errors. So once they go down a path they can’t properly backtrack. This feels like the ground truth I’ve experienced in LLMs to date.

> So once they go down a path they can’t properly backtrack.

That's what the specific training in o1 / r1 / qwq are addressing. The model outputs things like "i need to ... > thought 1 > ... > wait that's wrong > i need to go back > thought 2 > ... etc

Re: Procedural knowledge in pretraining drives reasoning in large language models

#15

Post by scellus »

jpcom wrote: post #9
mattdeboard wrote: post #8

Earlier quoted context omitted.

No. If I'm understanding correctly it means the software is learning how to solve problems in general by ingesting examples of procedural problem-solving.

You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem. The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how…

Yes, except that I'm not so sure there is a clear distinction between following general instructions and generating new heuristics. It's just a difference in the level of abstraction there, and probably not even that one in any discrete sense, more like a continuum.

(Current) models may of course lack sufficient training data to act on a metalevel enough ("be creative problem solvers"), or they may lack deep enough representations to efficiently act in a more creative way. (And those two may be more or less the same thing or not.)

Re: Procedural knowledge in pretraining drives reasoning in large language models

#17

Post by shermantanktop »

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be.

We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#18

Post by exe34 »

scellus wrote: post #15
jpcom wrote: post #9

Earlier quoted context omitted.

You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem. The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how…

Yes, except that I'm not so sure there is a clear distinction between following general instructions and generating new heuristics. It's just a difference in the level of abstraction there, and probably not even that one in any discrete sense, more like a continuum. (Current) models may of course lack sufficient training data to act on a metalevel enough ("be creative problem solvers"), or they may lack deep enough r…

it's exactly how we learn. many examples and then general principles. if you start with general principles, everybody drops out.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#19

Post by exe34 »

Retric wrote: post #13
jebarker wrote: post #12

Earlier quoted context omitted.

> In essence, it's like a neural net learning to follow a recipe by watching a chef cook—rather than inventing its own recipes entirely from first principles. Just like how a chef learns

A chef also learns through trial and error not just reading how others have cooked in the past and then copping their motions. This is exemplified by how altitude has a meaningful impact but isn’t discussed for a given recipe.

a text LLM isn't going to learn by trial and error, it's not been given that sort of freedom. RLHF would be the llm version of trial and error - but it's like the chef is only allowed to do that for a few days after years of chef school and from then on, he has to stick to what he has already learnt.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#20

Post by jebarker »

exe34 wrote: post #19
Retric wrote: post #13

Earlier quoted context omitted.

A chef also learns through trial and error not just reading how others have cooked in the past and then copping their motions. This is exemplified by how altitude has a meaningful impact but isn’t discussed for a given recipe.

a text LLM isn't going to learn by trial and error, it's not been given that sort of freedom. RLHF would be the llm version of trial and error - but it's like the chef is only allowed to do that for a few days after years of chef school and from then on, he has to stick to what he has already learnt.

Why isn't LLM pre-training based on next token prediction considered "trial and error"? It seems to fit that description pretty well to me.
Post reply on HN