Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Procedural knowledge in pretraining drives reasoning in large language models

arxiv.org

Post reply on HN
Print view View original

21–30 of 104 posts

Re: Procedural knowledge in pretraining drives reasoning in large language models

#21

Post by bravura »

exe34 wrote: post #18
scellus wrote: post #15

Earlier quoted context omitted.

Yes, except that I'm not so sure there is a clear distinction between following general instructions and generating new heuristics. It's just a difference in the level of abstraction there, and probably not even that one in any discrete sense, more like a continuum. (Current) models may of course lack sufficient training data to act on a metalevel enough ("be creative problem solvers"), or they may lack deep enough r…

it's exactly how we learn. many examples and then general principles. if you start with general principles, everybody drops out.

Not "exactly" how we learn. Humans learn through a combination of reinforcement learning (which is costly/risky/painful) and through observation of existing patterns and norms.

Better observation-based learning is a less expensive way of improving existing corpus-based approaches than trial-and-error and participating in an environment.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#22

Post by ninetyninenine »

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies

surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. Last time I was talking about LLM intelligence someone rudely told me to read up on how LLMs work and that we already know exactly how they work and they're just token predictors.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#23

Post by limit499karma »

jpcom wrote: post #9
mattdeboard wrote: post #8

Earlier quoted context omitted.

No. If I'm understanding correctly it means the software is learning how to solve problems in general by ingesting examples of procedural problem-solving.

You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem. The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how…

> it observes

Observe implies sentience that, without question, a neural net simply does not possess. "It" certainly 'records', or more specifically it 'maps', but there is no observer in sight (npi).

> mimic

LLM's do not mimic. The magic is mathematical and happening in the high dimensional space. If there is intrinsic underlying pattern and semantic affinities between process X (used in training) and process Y (used in application), it is very likely that both share proximity, possibly form, in some dimensions of the high dimensional model.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#24

Post by ChadNauseam »

jpcom wrote: post #9
mattdeboard wrote: post #8

Earlier quoted context omitted.

No. If I'm understanding correctly it means the software is learning how to solve problems in general by ingesting examples of procedural problem-solving.

You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem. The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how…

spoken eerily similar to how chatgpt would put it :) https://chatgpt.com/share/674cd11d-a30c-8005-90a3-023d0c9c18...

Re: Procedural knowledge in pretraining drives reasoning in large language models

#25

Post by strken »

ijk wrote: post #5

This would explain the unexpected benefits of training on code.

That sounds interesting, but I'm a layman and don't know anything about it. Can you provide a link?

I was able to find https://arxiv.org/abs/2408.10914, but I don't have the context to know whether it's the paper you're talking about.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#26

Post by rors »

It seems obvious to me that LLMs wouldn't be able to find examples of every single problem posed to them in training data. There wouldn't be enough examples for the factual look up needed in an information retrieval style search. I can believe that they're doing some form of extrapolation to create novel solutions to posed problems.

It's interesting that this paper doesn't contradict the conclusions of the Apple LLM paper[0], where prompts were corrupted to force the LLM into making errors. I can also believe that LLMs can only make small deviations from existing example solutions in creation of these novel solutions.

I hate that we're using the term "reasoning" for this solution generation process. It's a term coined by LLM companies to evoke an almost emotional response on how we talk about this technology. However, it does appear that we are capable of instructing machines to follow a series of steps using natural language, with some degree of ambiguity. That in of itself is a huge stride forward.

[0] https://machinelearning.apple.com/research/gsm-symbolic

Re: Procedural knowledge in pretraining drives reasoning in large language models

#28

Post by exe34 »

jebarker wrote: post #20
exe34 wrote: post #19

Earlier quoted context omitted.

a text LLM isn't going to learn by trial and error, it's not been given that sort of freedom. RLHF would be the llm version of trial and error - but it's like the chef is only allowed to do that for a few days after years of chef school and from then on, he has to stick to what he has already learnt.

Why isn't LLM pre-training based on next token prediction considered "trial and error"? It seems to fit that description pretty well to me.

a chef doesn't get feedback on his meal after picking up the spoon. he gets feedback when he or somebody else tastes the meal part way through and at the end.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#29

Post by exe34 »

bravura wrote: post #21
exe34 wrote: post #18

Earlier quoted context omitted.

it's exactly how we learn. many examples and then general principles. if you start with general principles, everybody drops out.

Not "exactly" how we learn. Humans learn through a combination of reinforcement learning (which is costly/risky/painful) and through observation of existing patterns and norms. Better observation-based learning is a less expensive way of improving existing corpus-based approaches than trial-and-error and participating in an environment.

except that the careful observation comes late in the curriculum. children don't learn if you start out with the Stern Gerlach experiment. they sing ABCs.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#30

Post by GarnetFloride »

ricardobeat wrote: post #27

Does this mean LLMs might do better if trained on large amounts of student notes, exams, book reviews and such? That would be incredibly interesting.

I have wondered that from time to time, why not train an AI system using educational curricula plus some games and play? It might be fascinating to see what comes out using various systems from around the world.
Post reply on HN