Dispelling misconceptions about RLHF
aerial-toothpaste-34a.notion.site
Dispelling misconceptions about RLHF
1–10 of 35 posts
Re: Dispelling misconceptions about RLHF
#2[deleted]
Re: Dispelling misconceptions about RLHF
#3Everything the post says about the behaviour of OpenAI models seems to be based on pure speculation.
Re: Dispelling misconceptions about RLHF
#4Everything the post says about the behaviour of OpenAI models seems to be based on pure speculation.
Yeah, in my opinion you can just skip that part and go straight to the author's description of failing to train their own model at first and what they ended up changing to make it work: https://aerial-toothpaste-34a.notion.site/How-OpenAI-Misled-...
Re: Dispelling misconceptions about RLHF
#5Another way to do reinforcement learning is to train a model to judge the quality of its own answers, to match judgements from experts or synthetically created. Until it develops an ability to judge its answer quality even if it can’t yet use that information to improve its responses.
It can be easier to recognize good responses than generate them.
Then feed it queries, generating its responses and judgements. Instead of training the responses to match response data, train it to output a high positive judgement, but while holding its “judgment” weight values constant. To improve its judgement values, the model is now being trained to give better answers since the judgment weights being back propagated act as a distributor of information from judgement back to how the responses should change to improve.
Learn to predict/judge what is good or bad. Then learn to maximize good and minimize bad using the judgment/prediction as a proxy for actual feedback.
This technique is closer to traditional human/animal reinforcement learning.
How we learn to predict situations that will cause us pain or positive affects, then learn to choose actions that minimize our predictions of bad, and maximize our predictions of good. Which is much more efficient way to learn than the expense of having to actually experience everything and always get explicit external feedback.
There are a many many ways to do reinforcement learning.
Re: Dispelling misconceptions about RLHF
#6Another way to do reinforcement learning is to train a model to judge the quality of its own answers, to match judgements from experts or synthetically created. Until it develops an ability to judge its answer quality even if it can’t yet use that information to improve its responses. It can be easier to recognize good responses than generate them. Then feed it queries, generating its responses and judgements. Instea…
The snag is: 'experts' aren’t neutral oracles. Many are underpaid and end up parroting whoever funds them. Lobby groups quietly buy authority all the time. So the real challenge isn’t just training on expert judgments, it’s making the model sharp enough to spot the BS in those judgments - otherwise you’re just encoding the bias straight into the weights.
Re: Dispelling misconceptions about RLHF
#7Another way to do reinforcement learning is to train a model to judge the quality of its own answers, to match judgements from experts or synthetically created. Until it develops an ability to judge its answer quality even if it can’t yet use that information to improve its responses. It can be easier to recognize good responses than generate them. Then feed it queries, generating its responses and judgements. Instea…
The snag is: 'experts' aren’t neutral oracles. Many are underpaid and end up parroting whoever funds them. Lobby groups quietly buy authority all the time. So the real challenge isn’t just training on expert judgments, it’s making the model sharp enough to spot the BS in those judgments - otherwise you’re just encoding the bias straight into the weights.
Which is why the foundation players must soon take on the additional role of being an ad buyer.
Interactive stuff, within content. A mini game in a game, school homework of course, or "whichever text box the viewer looks at longest by WorldCoin Eyeball Tracker for Democracy x Samsung" for an interstitial turned captcha.
Better hope your taste isn't too bland and derivative!
Amazon and Ali soon lap the field by allowing coupon farming, but somehow eventually end up where they started.
Re: Dispelling misconceptions about RLHF
#8this seems to disagree with a lot of research showing RL is not necessary for reasoning -- im not sure about alignment
Re: Dispelling misconceptions about RLHF
#9The mechanisms the author describe are used for RLHF, but are not sufficient for training the recent slew of “reasoning models.” To do that, you have to generate rewards not based on proximity to some reference full answer transcript, but rather based on how well the final answer (ex: the part after the “thinking tokens”) meets your reward criteria. This turns out to be a lot harder to do than the mechanisms used for RLHF which is one reason why we had RLHF for a while before we got the “reasoning models.” It’s also the only way you can understand the Sutskever quote “You’ll know your RL is working when the thinking tokens are no longer English” (a paraphrase, pulled from my memory).
Re: Dispelling misconceptions about RLHF
#10I'm confused about their definition of RL.
> ... SFT is a subset of RL.
> The first thing to note about traditional SFT is that the responses in the examples are typically human written. ... But it is also possible to build the dataset using responses from the model we’re about to train. ... This is called Rejection Sampling.
I can see why someone might say there's overlap between RL and SFT (or semi-supervised FT), but how is "traditional" SFT considered RL? What is not RL then? Are they saying all supervised learning is a subset of RL, or only if it's fine tuning?