Training Language Models to Self-Correct via Reinforcement Learning
1–10 of 95 posts
Re: Training Language Models to Self-Correct via Reinforcement Learning
#2It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1).
I don't see any mention of weight release unfortunately.
Re: Training Language Models to Self-Correct via Reinforcement Learning
#3Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law).
The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).
Re: Training Language Models to Self-Correct via Reinforcement Learning
#4Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).
Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.
Re: Training Language Models to Self-Correct via Reinforcement Learning
#5Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).
Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.
“Label bias” or “observation bias” a phenomenon where going outside of the learned path lives little room for error correction. Lecun talks about the lack of joint learning in LLMs.
Re: Training Language Models to Self-Correct via Reinforcement Learning
#6Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).
Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.
A reference could be this:
https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-la...
(Speaking of "law" is rhetoric, but an idea is pretty clear.)
Re: Training Language Models to Self-Correct via Reinforcement Learning
#7Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).
If you're talking about label bias then you don't need to solve label bias to 'solve' hallucinations when the model has already learnt internally when it's bullshitting or going off the rails.
Re: Training Language Models to Self-Correct via Reinforcement Learning
#8Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).
Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.
I think the OP may be referring to this slide that Yann LeCun has presented on several occasions:
https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901
To quote from the slide:
* Probability e that any produced token takes us outside the set of correct answers
* Probability that answer of length n is correct
* P(correct) = (1-e)^n
* This diverges exponentially
* It's not fixable (without a major redesign)Re: Training Language Models to Self-Correct via Reinforcement Learning
#9It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1). I don't see any mention of weight release unfortunately.
how is it similar?
Re: Training Language Models to Self-Correct via Reinforcement Learning
#10Earlier quoted context omitted.
Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.
I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)
Doesn't that argument make the fundamentally incorrect assumption that the space of produced output sequence has pockets where all output sequence with a certain prefix are incorrect?
Design your output space in such way that every prefix has a correct completion and this simplistic argument no longer applies. Humans do this in practice by saying "hold on, I was wrong, here's what's right".
Of course, there's still a question of whether you can get the probability mass of correct outputs large enough.