Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Procedural knowledge in pretraining drives reasoning in large language models

arxiv.org

Post reply on HN
Print view View original

81–90 of 104 posts

Re: Procedural knowledge in pretraining drives reasoning in large language models

#81

Post by naasking »

ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

I don't think "next token predictor" and "intelligent" are actually mutually exclusive.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#82

Post by naasking »

vundercind wrote: post #41
ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but about some often-correlated but different set of facts about relationships between tokens in writing. It’s the failure modes that make the distinction clearest. LLM output is only meaningful, in the way we usually mean that, at the point we assigned external, hum…

> or, at least, not “reasoning” about the things a human would be to solve the problems

You can't actually infer that either. Humans have considerable context that LLMs lack. You have no basis to infer how a human would reason given the same context as an LLM, or vice versa.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#83

Post by naasking »

ilaksh wrote: post #65
GarnetFloride wrote: post #30

Earlier quoted context omitted.

I have wondered that from time to time, why not train an AI system using educational curricula plus some games and play? It might be fascinating to see what comes out using various systems from around the world.

They do train on textbooks.

Yes, but maybe they should train more on uncertainty and mistakes, that are later followed by clarifications and corrections, like those found in notes.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#84

Post by mistermann »

eru wrote: post #54
patcon wrote: post #44

Earlier quoted context omitted.

> Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up Disagree that it's easy to pin on "how they were brought up". It seems very likely that we may learn that the flaws are part of what makes our intelligence "work" and be adaptive to changing environments. It may be favourable in terms of cultural evolution for parents t…

> Disagree that it's easy to pin on "how they were brought up". Indeed. That might play a role, but another less politically charged aspect to look at is just: how much effort is the human currently putting in? Humans are often on autopilot, perhaps even most of the time. Autopilot means taking lazy intellectual shortcuts. And to echo your argument: in familiar environments those shortcuts are often a good idea! If y…

Climate change and war are excellent examples demonstrating how far Humans are willing/obligated to take this convention.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#85

Post by vundercind »

naasking wrote: post #82
vundercind wrote: post #41

Earlier quoted context omitted.

The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but about some often-correlated but different set of facts about relationships between tokens in writing. It’s the failure modes that make the distinction clearest. LLM output is only meaningful, in the way we usually mean that, at the point we assigned external, hum…

> or, at least, not “reasoning” about the things a human would be to solve the problems You can't actually infer that either. Humans have considerable context that LLMs lack. You have no basis to infer how a human would reason given the same context as an LLM, or vice versa.

I don't think a human could effectively "reason" after being trained on nonsense (I don't think the training would even take). I think believing generative AI is operating on the same kind of meaning we are is a good way to be surprised when they go from writing like a learned professor for paragraphs to suddenly writing in the same tone and confidence but entirely wrong and with a bunch of made-up crap—it's all made-up crap from their perspective (if you will), we've just guided them into often making up crap that correlates to things we, separately from them, regard (they don't "regard", not in this sense) as non-crap.

[EDIT] To put it another way: if these things were trained to, I dunno, generate strings of onomatopoeia animal and environmental noises, I don't think anybody would be confusing what they're doing with anything terribly similar to human cognition, even if the output were structured and followed on from prompts reasonably sensibly and we were able to often find something like meaning or mood or movement or locality in the output—but they'd be doing exactly the same thing they're doing now. I think the form of the output and the training sets we've chosen are what're making people believe they're doing stuff significantly like thinking, but it's all the same to an LLM.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#86

Post by ninetyninenine »

vundercind wrote: post #85
naasking wrote: post #82

Earlier quoted context omitted.

> or, at least, not “reasoning” about the things a human would be to solve the problems You can't actually infer that either. Humans have considerable context that LLMs lack. You have no basis to infer how a human would reason given the same context as an LLM, or vice versa.

I don't think a human could effectively "reason" after being trained on nonsense (I don't think the training would even take). I think believing generative AI is operating on the same kind of meaning we are is a good way to be surprised when they go from writing like a learned professor for paragraphs to suddenly writing in the same tone and confidence but entirely wrong and with a bunch of made-up crap—it's all made…

But how can you be sure? You talk with confidence as if evidence exists to prove what you say but none of this evidence exists. It’s almost as if you’re an LLM yourself making up a claim with zero evidence. Sure you have examples that correlate with your point but nothing that proves your point.

Additionally there exists LLM output that runs counter to your point. Explain LLM output that is correct and novel. There exists correct LLM output on queries that are so novel and unique they don’t exist in any form in the training data. You can easily and I mean really easily make an LLM produce such output.

Again you’re making up your answer here without proof or evidence which is identical to the extrapolation the LLM does. And your answer runs counter to every academic author on that paper. So what I don’t understand from people like you is the level of unhinged confidence that runs border to religion.

Like you were talking about how the wrongness of certain LLM output make the distinction clearest while obviously ignoring the output that makes it unclear.

It’s utterly trivial to get LLMs to output things that disprove your point. But what’s more insane is that you can get LLMs to explain all of what’s being debated in this thread to you.

https://chatgpt.com/share/674dd1fa-4934-8001-bbda-40fe369074...

Re: Procedural knowledge in pretraining drives reasoning in large language models

#87

Post by ninetyninenine »

naasking wrote: post #81
ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

I don't think "next token predictor" and "intelligent" are actually mutually exclusive.

I agree. But then why do so many people take this view point? I mean we all can guess as to why but I want to hear the reasoning from them.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#88

Post by ninetyninenine »

moffkalast wrote: post #77
ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

The reality is that both can be true at the same time. Yes they're next token predictors, but sometimes the only way to do that correctly is by actually understanding everything that came before and reasoning logically about it. There's some Sutskever quote that if the input to a model is most of a crime novel, and the next token is the name of the perpetrator, then the model understood the novel. Transformers are ar…

Perhaps reasoning and understanding are things that don’t exist and that humans themselves are just next token predictors.

Heck reasoning and understanding are ill defined concepts. It might even be running on the same bullshit fuel as the word “spirituality”. Maybe the rigorous definition of all these vague words like intelligence or comprehension is really just next token prediction.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#89

Post by moffkalast »

ninetyninenine wrote: post #88
moffkalast wrote: post #77

Earlier quoted context omitted.

The reality is that both can be true at the same time. Yes they're next token predictors, but sometimes the only way to do that correctly is by actually understanding everything that came before and reasoning logically about it. There's some Sutskever quote that if the input to a model is most of a crime novel, and the next token is the name of the perpetrator, then the model understood the novel. Transformers are ar…

Perhaps reasoning and understanding are things that don’t exist and that humans themselves are just next token predictors. Heck reasoning and understanding are ill defined concepts. It might even be running on the same bullshit fuel as the word “spirituality”. Maybe the rigorous definition of all these vague words like intelligence or comprehension is really just next token prediction.

Eh well they were defined in the world where human intelligence was the only thing around, so there were... baked in assumptions. But as the current definitions go, I don't think there's any need for included consciousness or self awareness for these terms to apply, despite what people seem to immediately jump to. There's no part where it needs to mean human-level proficiency either.

Understanding is having knowledge about something, reasoning is making logical conclusions from it, and intelligence is being able to do it continuously with newly presented information. All things that well trained LLMs can arguably do to a detectable extent.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#90

Post by Retric »

isaacfrond wrote: post #78
Jensson wrote: post #48

Earlier quoted context omitted.

There is more than one correct answer in reality, LLM pre-training just trains it to respond the same way as the text did. Imagine if school only gave correct if you used exactly the same words as the book, that is not "trial and error".

I can tell you haven't been in a school in while. That is actually a pretty accurate description of what schools are like nowadays.

Pretty accurate != always, which is the point.
Post reply on HN