Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Procedural knowledge in pretraining drives reasoning in large language models

arxiv.org

Post reply on HN
Print view View original

41–50 of 104 posts

Re: Procedural knowledge in pretraining drives reasoning in large language models

#41

Post by vundercind »

ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but about some often-correlated but different set of facts about relationships between tokens in writing.

It’s the failure modes that make the distinction clearest.

LLM output is only meaningful, in the way we usually mean that, at the point we assigned external, human meaning to it, after the fact. The LLM wouldn’t stop operating or become “confused” if fed gibberish, because the meaning it’s extracting doesn’t depend on the meaning humans assign things, except by coincidence—which coincidence we foster by feeding them with things we do not regard as gibberish, but that’s beside the point so far as how they “really work” goes.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#42

Post by pfisherman »

exe34 wrote: post #29
bravura wrote: post #21

Earlier quoted context omitted.

Not "exactly" how we learn. Humans learn through a combination of reinforcement learning (which is costly/risky/painful) and through observation of existing patterns and norms. Better observation-based learning is a less expensive way of improving existing corpus-based approaches than trial-and-error and participating in an environment.

except that the careful observation comes late in the curriculum. children don't learn if you start out with the Stern Gerlach experiment. they sing ABCs.

The parent of any young child can tell you that they learn through lots of exploration and reinforcement - often to the worry and chagrin of caregivers. Indeed much of our job is to guide exploration away from excessively dangerous “research” activities (ex. locking away cleaning products).

Re: Procedural knowledge in pretraining drives reasoning in large language models

#43

Post by MurizS »

strken wrote: post #25
ijk wrote: post #5

This would explain the unexpected benefits of training on code.

That sounds interesting, but I'm a layman and don't know anything about it. Can you provide a link? I was able to find https://arxiv.org/abs/2408.10914 , but I don't have the context to know whether it's the paper you're talking about.

I think GP was probably referring to "Scaling Data-Constrained Language Models" (2305.16264) from NeurIPS 2023, which looked first at how to optimally scale LLMs when training data is limited. There is a short section on mixing code (Python) into the training data and the effect this has on performance on e.g. natural language tasks. One of their findings was that training data can be up to 50% code without actually degrading performance, and in some cases (benchmarks like bAbI and WebNLG) with improvements (probably because these tasks have an emphasis on what they call "long-range state tracking capabilities").

For reference: In the Llama 3 technical report (2407.21783), they mention that they ended up using 17% code tokens in their training data.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#44

Post by patcon »

sigmoid10 wrote: post #37
shermantanktop wrote: post #17

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be. We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is sprea…

> Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up

Disagree that it's easy to pin on "how they were brought up". It seems very likely that we may learn that the flaws are part of what makes our intelligence "work" and be adaptive to changing environments. It may be favourable in terms of cultural evolution for parents to indoctrinate flawed logic, not unlike how replication errors are part of how evolution can and must work.

In other words: I'm not sure these "failures" of the models are actual failures (in the sense of being non-adaptive and important to the evolutionary processes of intelligence), and further, it is perhaps us humans that are "failing" by over-indexing on "reason" as explanation for how we arrived here and continue to persist in time ;)

Re: Procedural knowledge in pretraining drives reasoning in large language models

#46

Post by pclmulqdq »

sigmoid10 wrote: post #39
pclmulqdq wrote: post #38

Earlier quoted context omitted.

It's because we can put responsibility on humans to be correct but we can't on computers. Humans given the appropriate incentives are very good at their jobs and there is a path for compensation if they screw up. Computers have neither of these things.

Humans already put a lot of trust in computers not because they can take responsibility but because traditional software can be made very predictable or at least compliant. There are whole industries built around software standards to ensure that. The problem is we don't yet know enough about identifying and patching problems in these models. Once we get something equivalent to MISRA for LLMs to achieve the same leve…

Yes. Traditional software has an unbroken chain of hard responsibility back to a human. Once you introduce non-determinism, things get weird.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#47

Post by cscurmudgeon »

shermantanktop wrote: post #17

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be. We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

> We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

I never understood this line of reasoning.

1. Humans can't run faster than 30 mph.

2. Therefore we can't complain if cars/trains/transport always go slower than 30 mph.

These comparisons also hide that we are comparing best of AI (massive LLMs) with median/average human reasoning.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#48

Post by Jensson »

jebarker wrote: post #20
exe34 wrote: post #19

Earlier quoted context omitted.

a text LLM isn't going to learn by trial and error, it's not been given that sort of freedom. RLHF would be the llm version of trial and error - but it's like the chef is only allowed to do that for a few days after years of chef school and from then on, he has to stick to what he has already learnt.

Why isn't LLM pre-training based on next token prediction considered "trial and error"? It seems to fit that description pretty well to me.

There is more than one correct answer in reality, LLM pre-training just trains it to respond the same way as the text did.

Imagine if school only gave correct if you used exactly the same words as the book, that is not "trial and error".

Re: Procedural knowledge in pretraining drives reasoning in large language models

#49

Post by ninetyninenine »

vundercind wrote: post #41
ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but about some often-correlated but different set of facts about relationships between tokens in writing. It’s the failure modes that make the distinction clearest. LLM output is only meaningful, in the way we usually mean that, at the point we assigned external, hum…

But you also conveniently ignore the success modes where the answer is too novel to be anything other than reasoning.

The op clearly said LLMs reason so your opinion is totally against and opposed to the opinion of every author of that academic paper.

Why aren’t you condemning this paper?

Re: Procedural knowledge in pretraining drives reasoning in large language models

#50

Post by XenophileJKO »

cscurmudgeon wrote: post #47
shermantanktop wrote: post #17

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be. We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

> We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning. I never understood this line of reasoning. 1. Humans can't run faster than 30 mph. 2. Therefore we can't complain if cars/trains/transport always go slower than 30 mph. These comparisons also hide that we are comparing best of AI (massive LLMs) with median/average…

The whole social dynamic of this conversation is amazing. How fast complacency happened. In 2010 if you told me I could get a model to respond approximately as intelligently as a low intelligence human, I would be amazed. As a matter of perspective,I am still amazed.

At the same time I see such negative sentiment around the capabilities at their current limits.

We are reaching an era of commodified intelligence, which will be disruptive and surprising. Even the current limited models change the economics dramatically.

Post reply on HN