Dispelling misconceptions about RLHF • Hacker News Board

Re: Dispelling misconceptions about RLHF

#31

Post by JoshPurtell » Sun, Aug 17, 2025, 10:14 PM UTC

williamtrask wrote: post #13

Nit: the author says that supervised fine tuning is a type of RL, but it is not. RL is about delayed reward. Supervised fine tuning is not in any way about delayed reward.

RL is not about delayed reward. Multi-armed bandit problems have no credit assignment component, but are often the first RL problem taught.

In its most general, RL is about learning a policy (state -> action mapping). Which often requires inferring value, etc.

But copying a strong reference policy ... is still learning a policy. Whether by SFT or not

Re: Dispelling misconceptions about RLHF

#32

Post by Nevermark » Mon, Aug 18, 2025, 1:05 AM UTC

varispeed wrote: post #6

Nevermark wrote: post #5

Another way to do reinforcement learning is to train a model to judge the quality of its own answers, to match judgements from experts or synthetically created. Until it develops an ability to judge its answer quality even if it can’t yet use that information to improve its responses. It can be easier to recognize good responses than generate them. Then feed it queries, generating its responses and judgements. Instea…

The snag is: 'experts' aren’t neutral oracles. Many are underpaid and end up parroting whoever funds them. Lobby groups quietly buy authority all the time. So the real challenge isn’t just training on expert judgments, it’s making the model sharp enough to spot the BS in those judgments - otherwise you’re just encoding the bias straight into the weights.

> The snag is: 'experts' aren’t neutral oracles.

Without knowing who/what the experts are, how they are used, what they are judging, what structure and mitigations are in place around their use, and what degree of neutrality is required - with all other factors and techniques being used - you can't make any such claim.

It's so easy to dismiss something.

A general algorithm isn't a claim that its practical use won't require accommodating the specific complications of each context.

Very much like how data scientists don't expect their best algorithms to operate well, without also resolving a stream of practical issues. In standard and ad hoc ways, as needed.

Re: Dispelling misconceptions about RLHF

#33

Post by thinkzilla » Mon, Aug 18, 2025, 5:59 AM UTC

While the post uses DPO to illustrate RL and RLHF, in fact DPO is an alternative to RLHF that does not use RL. See the abstract of the DPO paper https://arxiv.org/abs/2305.18290, and Figure 1 in the paper: "DPO optimizes for human preferences while avoiding reinforcement learning".

The confusion is understandable. The definition of RL in the Sutton/Barto book extends over two chapters iirc, and after reading it I did not see how it differed from other learning methods. Studying some of the academic papers cleared things up.

Re: Dispelling misconceptions about RLHF

#34

Post by krackers » Thu, Aug 21, 2025, 6:05 AM UTC

markisus wrote: post #27

pas wrote: post #25

Earlier quoted context omitted.

sorry, could you explain why is it harder, where the complexity creeps in (compared to some naive "pattern matching the end of the response" tactic)? thanks!

Just speculating but proximity to a reference answer is a much denser reward signal. In contrast, parsing out a final answer into a pass/fail only provides a sparse reward signal.

Yup, RLVR as implemented by Deepseek et al. use only outcome supervision instead of process supervision. There have been attempts to do process supervision though.

Re: Dispelling misconceptions about RLHF

#35

Post by krackers » Thu, Aug 21, 2025, 6:16 AM UTC

thinkzilla wrote: post #33

While the post uses DPO to illustrate RL and RLHF, in fact DPO is an alternative to RLHF that does not use RL. See the abstract of the DPO paper https://arxiv.org/abs/2305.18290 , and Figure 1 in the paper: "DPO optimizes for human preferences while avoiding reinforcement learning". The confusion is understandable. The definition of RL in the Sutton/Barto book extends over two chapters iirc, and after reading it I di…

I think there was some quote from Karpathy who said that RLHF isn't actually "true" RL. As an armchair person, even after trying to understand it RLHF always seemed so roundabout. You don't have some open ended environment, you already have a fixed set of preferences. Instead of directly optimizing the model against that like DPO does, RLHF goes out of its way to train value/reward networks encoding these preferences then optimizing against that. I assumed that maybe it was just done this way for performance or stability or some other math -heavy reason, it was good to see that my suspicion was not off-base.