Procedural knowledge in pretraining drives reasoning in large language models
1–10 of 104 posts
Re: Procedural knowledge in pretraining drives reasoning in large language models
#2Is this conclusion similar to my layman's understanding of AlphaGo vs AlphaZero? That human procedural knowledge helps ML training to a point, and from there on becomes a limitation?
Re: Procedural knowledge in pretraining drives reasoning in large language models
#3drives retrieval of patterns of procedure?
I mean - like for arithmetic?
Re: Procedural knowledge in pretraining drives reasoning in large language models
#4Is this conclusion similar to my layman's understanding of AlphaGo vs AlphaZero? That human procedural knowledge helps ML training to a point, and from there on becomes a limitation?
No. They're saying that the model they analyzed used mainly information on _how_ to solve math problems from its training data, rather than documents that contained the answers to the (identical) math problems:
> "We investigate which data influence the model’s produced reasoning traces and how those data relate to the specific problems being addressed. Are models simply ‘retrieving’ answers from previously seen pretraining data and reassembling them, or are they employing a more robust strategy for generalisation?"
> "When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning."
Example reasoning question: > "Prompt Calculate the answer: (7 - 4) * 7 Think step-by-step."
Re: Procedural knowledge in pretraining drives reasoning in large language models
#5This would explain the unexpected benefits of training on code.
Re: Procedural knowledge in pretraining drives reasoning in large language models
#6Is this conclusion similar to my layman's understanding of AlphaGo vs AlphaZero? That human procedural knowledge helps ML training to a point, and from there on becomes a limitation?
No. They're saying that the model they analyzed used mainly information on _how_ to solve math problems from its training data, rather than documents that contained the answers to the (identical) math problems: > "We investigate which data influence the model’s produced reasoning traces and how those data relate to the specific problems being addressed. Are models simply ‘retrieving’ answers from previously seen pret…
What I further got from this is the models are learning the methods, but not evaluating themselves along the way. They don’t check for errors.
So once they go down a path they can’t properly backtrack.
This feels like the ground truth I’ve experienced in LLMs to date.
Re: Procedural knowledge in pretraining drives reasoning in large language models
#7You mean you need humans to step-by-step solve a problem so a neural net can mimic it? It sounds kinda obvious now that I write it out.
Re: Procedural knowledge in pretraining drives reasoning in large language models
#8You mean you need humans to step-by-step solve a problem so a neural net can mimic it? It sounds kinda obvious now that I write it out.
No. If I'm understanding correctly it means the software is learning how to solve problems in general by ingesting examples of procedural problem-solving.
Re: Procedural knowledge in pretraining drives reasoning in large language models
#9You mean you need humans to step-by-step solve a problem so a neural net can mimic it? It sounds kinda obvious now that I write it out.
No. If I'm understanding correctly it means the software is learning how to solve problems in general by ingesting examples of procedural problem-solving.
You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem.
The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how humans solve problems procedurally and uses that to replicate similar reasoning. This is crucial because the step-by-step demonstrations give the model structure and guidance, which is different from learning a generalizable strategy for solving any kind of problem without those examples.
In essence, it's like a neural net learning to follow a recipe by watching a chef cook—rather than inventing its own recipes entirely from first principles.
Re: Procedural knowledge in pretraining drives reasoning in large language models
#10that resonates - less facts and more reasoning training data. The most low hanging in terms of non synthetic data probably being mathematical proofs. With prolog and the like many alternate reasoning paths could be generated. It's hard to say if these many-path would help in llm training without access to the gigantic machines (it's so unfair) to try it on.