Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training Language Models to Self-Correct via Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Training Language Models to Self-Correct via Reinforcement Learning

#71

Post by slashdave »

sharemywin wrote: post #20
YeGoblynQueenne wrote: post #8

Earlier quoted context omitted.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

Wouldn't this apply to all prediction machines that make errors. Humans make bad predictions all the time but we still seem to manage to do some cool stuff here and there. part of an agents architecture will be for it to minimize e and then ground the prediction loop against a reality check. making LLMs bigger gets you a lower e with scale of data and compute but you will still need it to check against reality. test…

Humans self-correct (they can push the delete button)

Re: Training Language Models to Self-Correct via Reinforcement Learning

#72

Post by psb217 »

pptr wrote: post #68
psb217 wrote: post #60

Earlier quoted context omitted.

"The difference between LLMs and other kinds of predictive models, or humans, is that those kinds of systems do not produce their output one token at a time, but all in one go, so their error basically stays constant." -- This is a big, unproven assumption. Any non-autoregressive model can be trivially converted to an autoregressive model by: (i) generating a full output sequence, (ii) removing all tokens except the…

The loop itself is claimed to be the problem. It doesn't matter whether you use an AR or non-AR model. They both have a certain error probability that gets amplified in each iteration.

The per token error of the non-AR model wrapped with MPC is no higher than the per token error of the non-AR model without MPC. Likelihood of the entire sequence being off the true data manifold is just one minus the product of the per token errors, whether or not you're running with the MPC loop. Ie, wrapping the non-AR model in an MPC loop and thereby converting it to an AR model (with a built-in planning mechanism) doesn't increase its probability of going off track.

Per token error compounding over sequence length happens whether or not the model's autoregressive. The way in which per token errors correlate across a sequence might be more favorable wrt probability of producing bad sequences if you incorporate some explicit planning mechanism -- like the non-AR model wrapped in an MPC loop, but that's a more subtle argument than LeCun makes.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#73

Post by slashdave »

YeGoblynQueenne wrote: post #8
plewd wrote: post #4

Earlier quoted context omitted.

Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

Simplistic, since it assumes probabilities are uncorrelated, when they clearly aren't. Also, there are many ways of writing the correct solution to a problem (you do not need to replicated an exact sequence of tokens).

Re: Training Language Models to Self-Correct via Reinforcement Learning

#74

Post by famouswaffles »

shawnz wrote: post #33
optimalsolver wrote: post #3

Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).

Does anyone here know, has anyone tried something like feeding the perplexity of previous tokens back into the model, so that it has a way of knowing when it's going off the rails? Maybe it could be trained to start responding less confidently in those cases, reducing its desire to hallucinate.

Models already know when they are going off the rails. https://news.ycombinator.com/item?id=41504226. That's not the problem. The problem is that they don't care to tell you.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#75

Post by psb217 »

drdeca wrote: post #69
psb217 wrote: post #60

Earlier quoted context omitted.

"The difference between LLMs and other kinds of predictive models, or humans, is that those kinds of systems do not produce their output one token at a time, but all in one go, so their error basically stays constant." -- This is a big, unproven assumption. Any non-autoregressive model can be trivially converted to an autoregressive model by: (i) generating a full output sequence, (ii) removing all tokens except the…

Could the argument be rescued by some additional assumptions? I agree with, and have previously also stated, the point you make there about “any non-auto-regressive model can be converted into an equivalent auto-regressive model by […]”, but, if one imposes additional restrictions on e.g. computation time, or something like that, I think that construction no longer works. Well, of course there are some additional ass…

I think it would be hard to make a solid argument that AR or non-AR is strictly better wrt full sequence error rates, whether or not we place constraints on compute, memory, etc. I'd guess that there's some intrinsic form of complexity inherent to any particular distribution of sequences which requires spending at least some amount of compute to achieve sequence generation error less than some epsilon. I'd also guess that AR and non-AR models could both achieve this bound in principle, though maybe it's practically harder with one or the other. It would be interesting to formally characterize this sort of complexity, but that's above my analytical pay grade.

The hash function example is interesting. I think the model could compute y prior to outputting any tokens and then output the `hash(y), y' sequence deterministically. In architectures like transformers, all the compute in earlier steps can be reused in later steps via attention, so it wouldn't be necessary to recompute y at each step as long as the model commits to a given y up front before starting to generate hash(y).

Re: Training Language Models to Self-Correct via Reinforcement Learning

#76

Post by whiplash451 »

qudat wrote: post #70
sensanaty wrote: post #34

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this…

> I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. I'd argue the opposite: people think a person's mind is in "deep thought" when it's actually just a ball of statistics.

Do you think that an LLM would spit out Latin and English if you trained it with homo sapiens mumbling?

Yet, humans managed to do that (albeit over many generations)

Ergo, humans are not just balls of statistics

Re: Training Language Models to Self-Correct via Reinforcement Learning

#77

Post by mdp2021 »

baq wrote: post #66
mdp2021 wrote: post #31

Earlier quoted context omitted.

According to which philologist? In short: they are both weak terms, 'hallucination' and 'confabulation', and we are using them in this context very loosely (and it should be in the open). About the terms themselves, "confabulate" means "exchanging stories", while "hallucinate" is less clear but probably means "to err". In psychiatry, "hallucinate" was apparently introduced by Esquirol and "confabulate" by Wernicke an…

Let's invent a new term, then! I propose: - digimagination - stochafubalation - statillucination - matmulshitting Let's ask chatgpt free (coz I'm cheap) what it not-thinks about these, and ask for more statistically generated bullshit: -- 8 These are some creative and fun suggestions! Here's my take: Digimagination: This one has a sleek, almost sci-fi ring to it. It captures the digital, machine-driven nature of lang…

My turn. What about "pretending"?

My judgement of yours: I love 'stochfabulating' and 'matmulshitting', but I condemn 'digimagining'. Digimagining is doable in proper terms (no pun intended).

Re: Training Language Models to Self-Correct via Reinforcement Learning

#78

Post by whiplash451 »

barbarr wrote: post #57
whimsicalism wrote: post #45

Earlier quoted context omitted.

LeCuns argument is seriously flawed. It is not at all a rigorous one and you should not make such sweeping statements based on nothing.

At this point I just invert everything LeCun says about AI. Chances are he'll flip flop on his own statement a few months later anyways.

LeCun has been pretty steady for years now.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#79

Post by baq »

mdp2021 wrote: post #77
baq wrote: post #66

Earlier quoted context omitted.

Let's invent a new term, then! I propose: - digimagination - stochafubalation - statillucination - matmulshitting Let's ask chatgpt free (coz I'm cheap) what it not-thinks about these, and ask for more statistically generated bullshit: -- 8 These are some creative and fun suggestions! Here's my take: Digimagination: This one has a sleek, almost sci-fi ring to it. It captures the digital, machine-driven nature of lang…

My turn. What about "pretending"? My judgement of yours: I love 'stochfabulating' and 'matmulshitting', but I condemn 'digimagining'. Digimagining is doable in proper terms (no pun intended).

"Pretending" is too human to my taste: it assumes the thing doing the pretend thing knows about the real thing. It's something kids do during play. I'm too afraid of the consequences to admit LLMs are anywhere near this situation wink

Re: Training Language Models to Self-Correct via Reinforcement Learning

#80

Post by Nevermark »

sensanaty wrote: post #34

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this…

The right word is "confabulation". Which is when we fill in missing information but may not be aware that we are doing it.

We all confabulate to some degree, as any neural system must, since no training data is stored perfectly.

Human "hallucinations" in contrast, are a particular kind of breakdown in our sensory feedback loops. Which is not a process LLMs even have.

Hallucinations occur when our internal sensory feedback loops overpower actual sensory input, resulting in a stream of false sensory experience/signals being generated and processed. The false running experience might still incorporate some actual sensory information or not.

When we dream, we are hallucinating - our sensory experience loop running free of our actual senses - to a productive purpose.

The reason our senses have feedback is so that we can use our interpretation of sensory input as cues to make interpreting the next moments input easier. But its important that our running interpretation can reset when new input significantly diverges from our expectations so it can quickly reorient.

(Not only is it important to revert to a raw input interpretation to ensure our running interpretation keeps up the actual context changes and corrects misinterpretations, but such resets signal that something novel or unexpected has happened, so likely trigger learning.)

So "hallucinations" was an unfortunate and misleading choice of terminology.

Post reply on HN