Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training Language Models to Self-Correct via Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

31–40 of 95 posts

Re: Training Language Models to Self-Correct via Reinforcement Learning

#31

Post by mdp2021 »

williamcotton wrote: post #16
plaguuuuuu wrote: post #11

LLMs have no direct recollection of the qualia of their own training. This is at least a major way that I self-correct myself: if I'm about to talk about something I know, I'll try and figure out how/why I know that thing and in so doing, try to gauge whether I actually know that thing, if I'm hallucinating, or if I actually heard it from a less than reliable source etc. I don't think LLMs can self-correct without re…

Unless you’re under the influence of something or having a severe mental health crisis you are not hallucinating, you’re confabulating.

According to which philologist? In short: they are both weak terms, 'hallucination' and 'confabulation', and we are using them in this context very loosely (and it should be in the open).

About the terms themselves, "confabulate" means "exchanging stories", while "hallucinate" is less clear but probably means "to err". In psychiatry, "hallucinate" was apparently introduced by Esquirol and "confabulate" by Wernicke and Bonhoeffer; neither concept seems to be akin to the substance of the phenomenon of "stochastic parrots bullshitting an unchecked narrative through formal plausibility".

See: "Hallucinations and related concepts - their conceptual background" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4515540/

and: "The Confabulating Mind: How the Brain Creates Reality" - https://psychiatryonline.org/doi/full/10.1176/appi.ajp.2008....

Re: Training Language Models to Self-Correct via Reinforcement Learning

#32

Post by throwawaymaths »

sharemywin wrote: post #20
YeGoblynQueenne wrote: post #8

Earlier quoted context omitted.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

Wouldn't this apply to all prediction machines that make errors. Humans make bad predictions all the time but we still seem to manage to do some cool stuff here and there. part of an agents architecture will be for it to minimize e and then ground the prediction loop against a reality check. making LLMs bigger gets you a lower e with scale of data and compute but you will still need it to check against reality. test…

No. Many prediction machines can give you a confidence value on the full outcome. By the nature of tokenization and the casual inference (you build a token one at a time, and they're not really semantically connected except in the kv cache lookups, which are generally hidden to the user), the confidence values are thrown out in practice and even a weak confidence value would be hard to retrieve.

I don't think it's impossible to obtain content with confidence assessments with the transformer architecture but maybe not in the way it's done now (like maybe another mayer on top).

Re: Training Language Models to Self-Correct via Reinforcement Learning

#33

Post by shawnz »

optimalsolver wrote: post #3

Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).

Does anyone here know, has anyone tried something like feeding the perplexity of previous tokens back into the model, so that it has a way of knowing when it's going off the rails? Maybe it could be trained to start responding less confidently in those cases, reducing its desire to hallucinate.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#34

Post by sensanaty »

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind.

No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this were a regular JSON API endpoint, you wouldn't say the API is hallucinating, you'd say "This API is shit" because it's broken.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#35

Post by reportgunner »

diggan wrote: post #15
roboboffin wrote: post #12

Earlier quoted context omitted.

Is this similar to the effect that I have seen when you have two different LLMs talking to each other, they tend to descend into nonsense ? A single error in one of the LLM's output and that then pushes the other LLM out of distribution. I kind of oscillatory effect when the train of tokens move further and further out of the distribution of correct tokens.

> Is this similar to the effect that I have seen when you have two different LLMs talking to each other, they tend to descend into nonsense ? Is that really true? I'd expect that with high temperature values, but otherwise I don't see why this would happen, and I've experimented with pitting same models against each other and also different models against different models, but haven't come across that particular prob…

Can you show examples ? In any AI related discussions there are only some claims by people and never examples of the AI working well.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#36

Post by frakt0x90 »

sensanaty wrote: post #34

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this…

Yeah it's simply model error. All models from Linear Regression to LLMs have error. I guess because this type of error is in the form of deceptively reasonable human language, it gets a different moniker. It's also notably harder to quantify so it might warrant a different name.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#37

Post by Philpax »

sensanaty wrote: post #34

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this…

Do we really need to have this discussion in every thread about LLMs?

Re: Training Language Models to Self-Correct via Reinforcement Learning

#38

Post by vjerancrnjak »

roboboffin wrote: post #12
YeGoblynQueenne wrote: post #8

Earlier quoted context omitted.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

Is this similar to the effect that I have seen when you have two different LLMs talking to each other, they tend to descend into nonsense ? A single error in one of the LLM's output and that then pushes the other LLM out of distribution. I kind of oscillatory effect when the train of tokens move further and further out of the distribution of correct tokens.

This is equivalent to the problem of maximum entropy Markov models and their application to sequence output.

After some point you’re conditioning your next decision on tokens that are severely out of the learned path and you don’t even see it’s that bad.

Usually this was fixed with cost sensitive learning or increased sampling of weird distributions during learning and then making the model learn to correct the mistake.

Another approach was to have an inference algorithm that maximize the output probability, but these algorithms are expensive (viterbi and other dynamic programming methods).

Feature modeling in NNs somewhat allowed us to ignore these issues and get good performance but they will show up again.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#39

Post by sensanaty »

Philpax wrote: post #37
sensanaty wrote: post #34

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this…

Do we really need to have this discussion in every thread about LLMs?

As long as AI-bros are pushing for making AI models seem like more than they are to pad their wallets, there'll be someone like me pointing out that, no, it's not "hallucinating", it's spitting bad data.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#40

Post by seydor »

sensanaty wrote: post #34

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this…

do you really want to have a discussion about 'thought' and 'mind'? i don't
Post reply on HN