Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training Language Models to Self-Correct via Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Training Language Models to Self-Correct via Reinforcement Learning

#41

Post by whimsicalism »

diggan wrote: post #14
elcomet wrote: post #2

It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1). I don't see any mention of weight release unfortunately.

I think this submission paper is talking about reinforcement learning as part of/after the main training, then the model does inference as normal. They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, it "thinks" with words and readjusts at runtime. At least that's my understanding from these two approac…

you are describing the same thing?

sorry as a practitioner i’m having trouble understanding what point/distinction you are trying to make

Re: Training Language Models to Self-Correct via Reinforcement Learning

#42

Post by whimsicalism »

sensanaty wrote: post #34

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this…

can we make a siloed version of HN for your political faction? it’s tiresome reading these in every thread

Re: Training Language Models to Self-Correct via Reinforcement Learning

#43

Post by whimsicalism »

sensanaty wrote: post #39
Philpax wrote: post #37

Earlier quoted context omitted.

Do we really need to have this discussion in every thread about LLMs?

As long as AI-bros are pushing for making AI models seem like more than they are to pad their wallets, there'll be someone like me pointing out that, no, it's not "hallucinating", it's spitting bad data.

I know lots of people working on AI. they are among the least bro-y group of people I have ever met.

There is simply nothing similar to actual bro-y finance culture among AI research engineers. It is entirely a figment of the media and backreaction that we currently have to portray everyone we don’t like as a “bro” - truth be damned.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#44

Post by josh-sematic »

diggan wrote: post #14
elcomet wrote: post #2

It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1). I don't see any mention of weight release unfortunately.

I think this submission paper is talking about reinforcement learning as part of/after the main training, then the model does inference as normal. They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, it "thinks" with words and readjusts at runtime. At least that's my understanding from these two approac…

They are indeed similar and OpenAI did indeed use RL at training time in a way that has not been done before, as does this approach. Yes both also involve some additional inference-time generation, but the problem is that (at least as of now) you can't get standard LLMs to actually do well with extra inference-time generation unless you have a training process that uses RL to teach them to do so effectively. I'm working on a blog post to explain more about this aimed at HN-level audiences. Stay tuned!

Re: Training Language Models to Self-Correct via Reinforcement Learning

#45

Post by whimsicalism »

optimalsolver wrote: post #3

Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).

LeCuns argument is seriously flawed. It is not at all a rigorous one and you should not make such sweeping statements based on nothing.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#46

Post by whimsicalism »

plewd wrote: post #4
optimalsolver wrote: post #3

Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).

Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.

It’s a thing in that he said it but it’s not an actual law and it has several obvious logical flaws. It applies just as equally to human utterances.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#47

Post by whimsicalism »

reportgunner wrote: post #35
diggan wrote: post #15

Earlier quoted context omitted.

> Is this similar to the effect that I have seen when you have two different LLMs talking to each other, they tend to descend into nonsense ? Is that really true? I'd expect that with high temperature values, but otherwise I don't see why this would happen, and I've experimented with pitting same models against each other and also different models against different models, but haven't come across that particular prob…

Can you show examples ? In any AI related discussions there are only some claims by people and never examples of the AI working well.

you’re saying you have never seen an example of AI working well?

Re: Training Language Models to Self-Correct via Reinforcement Learning

#48

Post by hackerlight »

vbarrielle wrote: post #26
littlestymaar wrote: post #23

Earlier quoted context omitted.

> * P(correct) = (1-e)^n * This diverges exponentially I don't get it, 1-e is between 0 and 1, so (1-e)^n converge to zero. Also, a probability cannot diverge since it's bounded by 1! I think the argument is that 1 - e^n converges to 1, which is what the law is about.

P(correct) converges to zero, so you get almost certainly incorrect, at an exponential rate. The original choice of terms is not the most rigorous, but the reasoning is sound (under the assumption that e is a constant).

P(correct) doesn't go down with token count if you have self-correction. It can actually go up with token count.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#49

Post by bithive123 »

sensanaty wrote: post #39
Philpax wrote: post #37

Earlier quoted context omitted.

Do we really need to have this discussion in every thread about LLMs?

As long as AI-bros are pushing for making AI models seem like more than they are to pad their wallets, there'll be someone like me pointing out that, no, it's not "hallucinating", it's spitting bad data.

You're being pedantic. Your statement that "it's spitting bad data" is incorrect too, as it implies agency. Actually, nothing is happening but electrons flowing. The notion of an "it" that "spits" "data" which is "bad" is your own conceptual overlay.

Re: Training Language Models to Self-Correct via Reinforcement Learning

#50

Post by mistrial9 »

whimsicalism wrote: post #43
sensanaty wrote: post #39

Earlier quoted context omitted.

As long as AI-bros are pushing for making AI models seem like more than they are to pad their wallets, there'll be someone like me pointing out that, no, it's not "hallucinating", it's spitting bad data.

I know lots of people working on AI. they are among the least bro-y group of people I have ever met. There is simply nothing similar to actual bro-y finance culture among AI research engineers. It is entirely a figment of the media and backreaction that we currently have to portray everyone we don’t like as a “bro” - truth be damned.

no - the cliques are different but linked at the hip. Add international finance, too.. India, China and others.
Post reply on HN