Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Procedural knowledge in pretraining drives reasoning in large language models

arxiv.org

Post reply on HN
Print view View original

51–60 of 104 posts

Re: Procedural knowledge in pretraining drives reasoning in large language models

#51

Post by eru »

pfisherman wrote: post #34
rors wrote: post #26

It seems obvious to me that LLMs wouldn't be able to find examples of every single problem posed to them in training data. There wouldn't be enough examples for the factual look up needed in an information retrieval style search. I can believe that they're doing some form of extrapolation to create novel solutions to posed problems. It's interesting that this paper doesn't contradict the conclusions of the Apple LLM…

I very much agree with the perspective that LLMs are not suited for “reasoning” in the sense of creative problem solving or application of logic. I think that the real potential in this domain is having them act as a sort of “compiler” layer that bridges the gap between natural language - which is imprecise - and formal languages (sql, prolog, python, lean, etc) that are more suited for solving these types of problem…

> I think that the real potential in this domain is having them act as a sort of “compiler” layer that bridges the gap between natural language - which is imprecise - and formal languages (sql, prolog, python, lean, etc) that are more suited for solving these types of problems. And then maybe synthesizing the results / outputs of the formal language layer. Basically “agents”.

Well, if you do all that, would you say that the system has a whole has 'reasoned'? (I think ChatGPT can already call out to Python.)

Re: Procedural knowledge in pretraining drives reasoning in large language models

#52

Post by eru »

sigmoid10 wrote: post #37
shermantanktop wrote: post #17

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be. We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is sprea…

> Most humans are not that good at reasoning themselves [...]

I'd say most humans most of the time. Individual humans can do a lot better (or worse) depending on how much effort they put in, and whether they slept well, had their morning coffee, etc.

> If anything, the current limits of these morels show the limits of human cognition which is spread throughout the internet - because this is literally what they learned from.

I wouldn't go quite so far. Especially because some tasks require smarts, even though there's no smarts in the training data.

The classic example is perhaps programming: the Python interpreter is not intelligent by any stretch of the imagination, but an LLM (or a human) needs smarts to predict what's going to do, especially if you are trying to get it to do something specific.

That example might skirt to close to the MuZero paper that you already mentioned as an exception / extension.

So let's go with a purer example: even the least smart human is a complicated system with a lot of hidden state, parts of that state shine through when that human produces text. Predicting the next token of text just from the previous text is a lot harder and requires a lot more smarts than if you had access to the internal state directly.

It's sort-of like an 'inverse problem'. https://en.wikipedia.org/wiki/Inverse_problem

Re: Procedural knowledge in pretraining drives reasoning in large language models

#53

Post by joe_the_user »

rors wrote: post #26

It seems obvious to me that LLMs wouldn't be able to find examples of every single problem posed to them in training data. There wouldn't be enough examples for the factual look up needed in an information retrieval style search. I can believe that they're doing some form of extrapolation to create novel solutions to posed problems. It's interesting that this paper doesn't contradict the conclusions of the Apple LLM…

I can believe that they're doing some form of extrapolation to create novel solutions to posed problems

You can believe it what sort of evidence are you using for this belief?

Edit: Also, the abstract of the Apple paper hardly says "corruption" (implying something tricky), it says that they changed the initial numerical values

Re: Procedural knowledge in pretraining drives reasoning in large language models

#54

Post by eru »

patcon wrote: post #44
sigmoid10 wrote: post #37

Earlier quoted context omitted.

This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is sprea…

> Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up Disagree that it's easy to pin on "how they were brought up". It seems very likely that we may learn that the flaws are part of what makes our intelligence "work" and be adaptive to changing environments. It may be favourable in terms of cultural evolution for parents t…

> Disagree that it's easy to pin on "how they were brought up".

Indeed. That might play a role, but another less politically charged aspect to look at is just: how much effort is the human currently putting in?

Humans are often on autopilot, perhaps even most of the time. Autopilot means taking lazy intellectual shortcuts. And to echo your argument: in familiar environments those shortcuts are often a good idea!

If you just do whatever worked last time you were in a similar situation, or whatever your peers are doing, chances are you'll have an easier time than reasoning everything out from scratch. Especially in any situations involving other humans cooperating with you, predictability itself is an asset.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#55

Post by eru »

pclmulqdq wrote: post #46
sigmoid10 wrote: post #39

Earlier quoted context omitted.

Humans already put a lot of trust in computers not because they can take responsibility but because traditional software can be made very predictable or at least compliant. There are whole industries built around software standards to ensure that. The problem is we don't yet know enough about identifying and patching problems in these models. Once we get something equivalent to MISRA for LLMs to achieve the same leve…

Yes. Traditional software has an unbroken chain of hard responsibility back to a human. Once you introduce non-determinism, things get weird.

Non-determinism is a well-understood tool, and does not diminish responsibility.

Grep works just fine, despite implementing non-deterministic finite state machines. Monte Carlo simulations are behind nuclear weapons (where they were invented), weather forecasts, financial trading, etc. Las Vegas algorithms like randomised quicksort also diminish no one's responsibility.

In principle, you can run training and inference on neural networks completely deterministically. But I don't think that makes any difference to responsibility. (To make them deterministic, you obviously have to use pseudo-random number generators with fixed seeds, but less obviously you also have to make sure that when you merge the results of parallel runs, the results 'merge' deterministically. Deterministic parallelism is an extremely interesting field of study! Or, since we are only talking about principles, not what's practical, you could just run everything in series.)

The problem with LLMs is that they are complicated and their actions are hard for humans to predict or reason through. Complexity is the bane of responsibility: if you have a complicated enough system (and a complicated enough management structure involved in producing that system), that's where responsibility goes to die, unless you specifically work to establish it.

In this case, employing LLMs is no worse than employing humans. If upper management gives bad instructions and incentives for lower level employees, we tend to pin the responsibility on upper management.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#56

Post by Nevermark »

spitfire wrote: post #11
spitfire wrote: post #6

Earlier quoted context omitted.

What I further got from this is the models are learning the methods, but not evaluating themselves along the way. They don’t check for errors. So once they go down a path they can’t properly backtrack. This feels like the ground truth I’ve experienced in LLMs to date.

I’ll add when I say “learning” I mean memorization. Memorizing on a higher level than facts. I would love to spend the time and see how altering the query alters the reasoning path. How firm is in the path once it’s chosen? A high level approach has the possibility to be very computer efficient.

> Memorizing on a higher level than facts

Which is not memorization, since memorization is defined by its limits: storing information based on its literal form, as apposed to some higher meaning.

It is called generalization. Learning specific examples with a shared memory too small to memorize all the examples, creates a gradient toward a more compact storage form: patterns. Which, unlike memorized examples, are able to generate reasonable guesses for similar but previously unencountered problems.

Generalization does not require reasoning, nor is it required for reasoning. But they often complement each other.

Where reasoning usually means some kind of flexible application of multiple steps. I.e. a sequence of steps, trying alternative steps, stepping forward to a solution, stepping back from the goal, accumulation of ever larger solved subsets or substeps of the problem, etc.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#57

Post by eru »

XenophileJKO wrote: post #50
cscurmudgeon wrote: post #47

Earlier quoted context omitted.

> We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning. I never understood this line of reasoning. 1. Humans can't run faster than 30 mph. 2. Therefore we can't complain if cars/trains/transport always go slower than 30 mph. These comparisons also hide that we are comparing best of AI (massive LLMs) with median/average…

The whole social dynamic of this conversation is amazing. How fast complacency happened. In 2010 if you told me I could get a model to respond approximately as intelligently as a low intelligence human, I would be amazed. As a matter of perspective,I am still amazed. At the same time I see such negative sentiment around the capabilities at their current limits. We are reaching an era of commodified intelligence, whic…

> Even the current limited models change the economics dramatically.

Yes, though at the moment they hype is still a lot bigger than the impact.

But I am fairly confident that even without any new technical ideas for the networks themselves, we will see a lot more economic impact over the next few years, as people work out how to use these new tools.

(Of course, the networks will also evolve still.)

Re: Procedural knowledge in pretraining drives reasoning in large language models

#58

Post by eru »

pfisherman wrote: post #42
exe34 wrote: post #29

Earlier quoted context omitted.

except that the careful observation comes late in the curriculum. children don't learn if you start out with the Stern Gerlach experiment. they sing ABCs.

The parent of any young child can tell you that they learn through lots of exploration and reinforcement - often to the worry and chagrin of caregivers. Indeed much of our job is to guide exploration away from excessively dangerous “research” activities (ex. locking away cleaning products).

As an ideal parent, you should give your kids access to activities that seem dangerous, without actually being all too dangerous.

Kids seem to have an internal dial for a desired level of perceived danger, and get up to weird stuff, if they don't get enough perceived danger.

Re: Procedural knowledge in pretraining drives reasoning in large language models

#59

Post by eru »

MurizS wrote: post #43
strken wrote: post #25

Earlier quoted context omitted.

That sounds interesting, but I'm a layman and don't know anything about it. Can you provide a link? I was able to find https://arxiv.org/abs/2408.10914 , but I don't have the context to know whether it's the paper you're talking about.

I think GP was probably referring to "Scaling Data-Constrained Language Models" (2305.16264) from NeurIPS 2023, which looked first at how to optimally scale LLMs when training data is limited. There is a short section on mixing code (Python) into the training data and the effect this has on performance on e.g. natural language tasks. One of their findings was that training data can be up to 50% code without actually…

Is the network only trained on the source code, or does it have access to the results of running the code, too?

Re: Procedural knowledge in pretraining drives reasoning in large language models

#60

Post by tkgally »

ben_w wrote: post #31
ninetyninenine wrote: post #22

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. La…

The loudest people seem to be those with the most extreme positions, and that includes on "is ${specific AI} (useless|superhuman) for ${domain}?". Perhaps it's just perception, but perhaps the arguments make them persist, as CGP Grey pointed out: https://www.youtube.com/watch?v=rE3j_RHkqJc As I'm in the middle, I get flack from people on both extremes, as I'm outside their (equivalent of or just literally?) Overton w…

> Makes me wonder what the real common discourse was of looms during the industrial revolution, and not just the modern summary of that era.

Interesting question. I did a little searching with help from Claude, ChatGPT, Google, and the Internet Archive. Here are some links to writings from that time:

“Thoughts on the use of machines, in the cotton manufacture” (1780)

https://archive.org/details/bim_eighteenth-century_thoughts-...

Excerpt: “How many writers and copiers of books were thrown out of employment, or obliged to change it, by the introduction of printing presses? About ten years ago, when the Spinning Jennies came up, old persons, children, and those who could not easily learn to use the new machines, did suffer, for a while; till families had learned to play into one another's hands, by each taking a different kind of work. But the general benefit, which was received from the machines, very soon silenced all objections. And every sensible man now looks upon them with gratitude and approbation. It is probable, this will be the case in all new inventions.”

Kevin Binfield, ed., Writings of the Luddites (1811-1815; 2004)

https://ia903409.us.archive.org/16/items/writings-of-the-lud...

Robert Owen, Observations on the effect of the manufacturing system (1817)

https://archive.org/details/observationsonef00owenrich/page/...

William Radcliffe, Origin of the new system of manufacture commonly called power-loom weaving (1828)

https://archive.org/details/originofnewsyste0000radc

“An address to the Glasgow cotton-spinners on the moral bearing of their association” (1838)

https://catalogue.nla.gov.au/catalog/6023196

Post reply on HN