Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Procedural knowledge in pretraining drives reasoning in large language models

arxiv.org

Post reply on HN
Print view View original

31–40 of 104 posts

Re: Procedural knowledge in pretraining drives reasoning in large language models

#31

Post by ben_w »

ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

The loudest people seem to be those with the most extreme positions, and that includes on "is ${specific AI} (useless|superhuman) for ${domain}?". Perhaps it's just perception, but perhaps the arguments make them persist, as CGP Grey pointed out: https://www.youtube.com/watch?v=rE3j_RHkqJc

As I'm in the middle, I get flack from people on both extremes, as I'm outside their (equivalent of or just literally?) Overton window on this subject. Seems like an odd zone to be in for the opinion "this is a useful tool, but I see loads of ways it can go wrong". Makes me wonder what the real common discourse was of looms during the industrial revolution, and not just the modern summary of that era.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#32

Post by btilly »

This is highly relevant to the recent discussion at https://news.ycombinator.com/item?id=42285128.

Google claims that their use of pretraining is a key requirement for being able to deliver a (slightly) better chip design. And they claim that a responding paper that did not attempt to do pretraining, should have been expected to be well below the state of the art in chip design.

Given how important reasoning is for chip design, and given how important pretraining is for driving reasoning in large language models, it is obvious that Google's reasoning is very reasonable. If Google barely beats the state of the art while using pretraining, an attempt that doesn't pretrain should be expected to be well below the current state of the art. And therefore that second attempt's poor performance says nothing about whether Google's results are plausible.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#33

Post by ucefkh »

rors wrote: post #26

It seems obvious to me that LLMs wouldn't be able to find examples of every single problem posed to them in training data. There wouldn't be enough examples for the factual look up needed in an information retrieval style search. I can believe that they're doing some form of extrapolation to create novel solutions to posed problems. It's interesting that this paper doesn't contradict the conclusions of the Apple LLM…

Totally, these companies are pushing towards showcasing their AI models as self thinking and reasoning AI while they are just trained of a lot of amount of data in dataset format which they extrapolate to find the right answer.

They still can't think outsider their box of datasets

Re: Procedural knowledge in pretraining drives reasoning in large language models

#34

Post by pfisherman »

rors wrote: post #26

It seems obvious to me that LLMs wouldn't be able to find examples of every single problem posed to them in training data. There wouldn't be enough examples for the factual look up needed in an information retrieval style search. I can believe that they're doing some form of extrapolation to create novel solutions to posed problems. It's interesting that this paper doesn't contradict the conclusions of the Apple LLM…

I very much agree with the perspective that LLMs are not suited for “reasoning” in the sense of creative problem solving or application of logic. I think that the real potential in this domain is having them act as a sort of “compiler” layer that bridges the gap between natural language - which is imprecise - and formal languages (sql, prolog, python, lean, etc) that are more suited for solving these types of problems. And then maybe synthesizing the results / outputs of the formal language layer. Basically “agents”.

That being said, I do think that LLMs are capable of “verbal reasoning” operations. I don’t have a good sense of the boundaries that distinguish the logics - verbal, qualitative, quantitative reasoning. What comes to my mind is the verbal sections of standardized tests.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#35

Post by unit149 »

jpcom wrote: post #9
mattdeboard wrote: post #8

Earlier quoted context omitted.

No. If I'm understanding correctly it means the software is learning how to solve problems in general by ingesting examples of procedural problem-solving.

You're close, but there’s an important nuance. The process isn't about "learning how to solve problems in general" in the broad sense. It's more specific: the neural network is trained to mimic the step-by-step process demonstrated by humans solving a specific problem. The distinction is that the software doesn't autonomously derive general problem-solving heuristics from scratch. Instead, it observes examples of how…

Crucially, this is what MacIntyre's narrativity thesis is talking about:

If a university professor is giving a lecture on decentralized finance and forks into a recipe for chocolate chip cookies: crack two eggs, add a cup of flour, and fold in brown sugar prior to baking, it would break linearity.

A generalizable strategy for synthesizing LLMs differentiated by their training parameters is a tokenization is isolating data sets and then establishing a lattice in uniformity within the field of technics.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#36

Post by pfisherman »

btilly wrote: post #32

This is highly relevant to the recent discussion at https://news.ycombinator.com/item?id=42285128 . Google claims that their use of pretraining is a key requirement for being able to deliver a (slightly) better chip design. And they claim that a responding paper that did not attempt to do pretraining, should have been expected to be well below the state of the art in chip design. Given how important reasoning is for…

I am not an expert in the particular application domain of that article; but I can see why their argument of pre training might be valid. It is not especially controversial to say that pre training neural nets improves few shot learning performance. And I suspect there is an inflection point for every problem where pre trained neural nets yield better few shot learning performance than less data hungry approaches - such as hand crafted features or strong priors.

That being said, it seems that the question here is whether that inflection point has been reached in this case.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#37

Post by sigmoid10 »

shermantanktop wrote: post #17

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be. We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is spread throughout the internet - because this is literally what they learned from. I believe once we achieve a more independent learning (like we've seen glimpses of in the MuZero paper) these models will blow human intelligence out of the water.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#38

Post by pclmulqdq »

sigmoid10 wrote: post #37
shermantanktop wrote: post #17

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be. We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is sprea…

It's because we can put responsibility on humans to be correct but we can't on computers. Humans given the appropriate incentives are very good at their jobs and there is a path for compensation if they screw up. Computers have neither of these things.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#39

Post by sigmoid10 »

pclmulqdq wrote: post #38
sigmoid10 wrote: post #37

Earlier quoted context omitted.

This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is sprea…

It's because we can put responsibility on humans to be correct but we can't on computers. Humans given the appropriate incentives are very good at their jobs and there is a path for compensation if they screw up. Computers have neither of these things.

Humans already put a lot of trust in computers not because they can take responsibility but because traditional software can be made very predictable or at least compliant. There are whole industries built around software standards to ensure that. The problem is we don't yet know enough about identifying and patching problems in these models. Once we get something equivalent to MISRA for LLMs to achieve the same level of compliance, there is very little that could still hold them back.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#40

Post by Retric »

jebarker wrote: post #20
exe34 wrote: post #19

Earlier quoted context omitted.

a text LLM isn't going to learn by trial and error, it's not been given that sort of freedom. RLHF would be the llm version of trial and error - but it's like the chef is only allowed to do that for a few days after years of chef school and from then on, he has to stick to what he has already learnt.

Why isn't LLM pre-training based on next token prediction considered "trial and error"? It seems to fit that description pretty well to me.

Pre-training is based on a proxy for desired output not actually desired output. It’s not in the form of responses to a prompt, and 1:1 reproducing copyrighted works in production would be bad.

It’s the difference between a painter copying some work and a painter making an original piece and then get feedback on it. We consider the second trial and error because the full process is being tested not just technique.

Post reply on HN