Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training Language Models to Self-Correct via Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Training Language Models to Self-Correct via Reinforcement Learning

#21

Post by sharemywin »

roboboffin wrote: post #12
YeGoblynQueenne wrote: post #8

Earlier quoted context omitted.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

Is this similar to the effect that I have seen when you have two different LLMs talking to each other, they tend to descend into nonsense ? A single error in one of the LLM's output and that then pushes the other LLM out of distribution. I kind of oscillatory effect when the train of tokens move further and further out of the distribution of correct tokens.

this is like the human game of telephone.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#22

Post by nsagent »

diggan wrote: post #14
elcomet wrote: post #2

It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1). I don't see any mention of weight release unfortunately.

I think this submission paper is talking about reinforcement learning as part of/after the main training, then the model does inference as normal. They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, it "thinks" with words and readjusts at runtime. At least that's my understanding from these two approac…

Both models generate an answer after multiple turns, where each turn has access to the outputs from a previous turn. Both refer to the chain of outputs as a trace.

Since OpenAI did not specify what exactly is in their reasoning trace, it's not clear what if any difference there is between the approaches. They could be vastly different, or they could be slight variations of each other. Without details from OpenAI, it's not currently possible to tell.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#23

Post by littlestymaar »

YeGoblynQueenne wrote: post #8
plewd wrote: post #4

Earlier quoted context omitted.

Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

> * P(correct) = (1-e)^n * This diverges exponentially

I don't get it, 1-e is between 0 and 1, so (1-e)^n converge to zero. Also, a probability cannot diverge since it's bounded by 1!

I think the argument is that 1 - e^n converges to 1, which is what the law is about.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#24

Post by wpietri »

QuadmasterXLII wrote: post #13
plaguuuuuu wrote: post #11

LLMs have no direct recollection of the qualia of their own training. This is at least a major way that I self-correct myself: if I'm about to talk about something I know, I'll try and figure out how/why I know that thing and in so doing, try to gauge whether I actually know that thing, if I'm hallucinating, or if I actually heard it from a less than reliable source etc. I don't think LLMs can self-correct without re…

So you’re saying the solution is to prefix each training batch with a description of a sensory experience (You read the following in a paris cafe in 1997. While you read, you have an excellent baguette and some boiled eggs, and over-roasted coffee. The woman one table over is wearing a beautiful blue hat) and then post-train the final model into recalling the setting where it read any piece of text, or failing to rec…

I don't think that's what they're saying at all. They're talking not about qualia in the human sense, but specifically about "the qualia of their own training". That is, the corpus that LLMs "learn" from and the "experiences" of those texts that are generalized during the training process. Both the raw data and the memory of "learning" is discarded.

So if one were to improve an LLM along those lines, I believe it would be something like: 1) LLM is asked a question. 2) LLM comes up with an initial response. 3) LLM retrieves the related "learning" history behind that answer and related portions of the corpus. 4) LLM compares the initial answer with the richer set of information, looking for conflicts between the initial answer and the broader set, or "learning" choices that may be false. 6) LLM generates a better answer and gives it. 7) LLM incorporates this new "learning".

And that strikes me as a pretty reasonable long-term approach, if not one that fits within the constraints of the current gold rush.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#25

Post by numeri »

diggan wrote: post #14
elcomet wrote: post #2

It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1). I don't see any mention of weight release unfortunately.

I think this submission paper is talking about reinforcement learning as part of/after the main training, then the model does inference as normal. They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, it "thinks" with words and readjusts at runtime. At least that's my understanding from these two approac…

OpenAI stated [1] that one of the breakthroughs needed for o1's train of thought to work was reinforcement learning to teach it to recover from faulty reasoning.

> Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses. It learns to recognize and correct its mistakes. It learns to break down tricky steps into simpler ones. It learns to try a different approach when the current one isn’t working.

That's incredibly similar to this paper, which is discusses the difficulty in finding a training method that guides the model to learn a self-correcting technique (in which subsequent attempts learn from and improve on previous attempts), instead of just "collapsing" into a mode of trying to get the answer right with the very first try.

[1]: https://openai.com/index/learning-to-reason-with-llms/

Re: Training Language Models to Self-Correct via Reinforcement Learning

#26

Post by vbarrielle »

littlestymaar wrote: post #23
YeGoblynQueenne wrote: post #8

Earlier quoted context omitted.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

> * P(correct) = (1-e)^n * This diverges exponentially I don't get it, 1-e is between 0 and 1, so (1-e)^n converge to zero. Also, a probability cannot diverge since it's bounded by 1! I think the argument is that 1 - e^n converges to 1, which is what the law is about.

P(correct) converges to zero, so you get almost certainly incorrect, at an exponential rate. The original choice of terms is not the most rigorous, but the reasoning is sound (under the assumption that e is a constant).

Re: Training Language Models to Self-Correct via Reinforcement Learning

#27

Post by vbarrielle »

ziofill wrote: post #19
YeGoblynQueenne wrote: post #8

Earlier quoted context omitted.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

Doesn’t this assume that the probability of a correct answer is iid? It can’t be that simple.

Yes the main flaw of this reasoning is supposing that e does not depend on previous output. I think this was a good approximation to characterize vanilla LLMs, but the kind of RL in this paper is done with the explicit goal of making e depending on prior output (and specifically to lower it given a long enough chain of thought).

Re: Training Language Models to Self-Correct via Reinforcement Learning

#28

Post by marcosdumay »

atq2119 wrote: post #10
YeGoblynQueenne wrote: post #8

Earlier quoted context omitted.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

Doesn't that argument make the fundamentally incorrect assumption that the space of produced output sequence has pockets where all output sequence with a certain prefix are incorrect? Design your output space in such way that every prefix has a correct completion and this simplistic argument no longer applies. Humans do this in practice by saying "hold on, I was wrong, here's what's right". Of course, there's still a…

How do you do this in something where the only memory is the last few things it said or heard?

Re: Training Language Models to Self-Correct via Reinforcement Learning

#29

Post by seydor »

optimalsolver wrote: post #3

Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).

"never" is not itself a problem, people do the same

you only need to solve fusion correctly once

Re: Training Language Models to Self-Correct via Reinforcement Learning

#30

Post by hackerlight »

YeGoblynQueenne wrote: post #8
plewd wrote: post #4

Earlier quoted context omitted.

Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.

I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)

It's quite fitting that the topic of this thread is self-correction. Self-correction is a trivial existence proof that refutes what LeCun is saying, because all the LLM has to say is "I made a mistake, let me start again".
Post reply on HN