Are alignment and fine-tuning just a parallel of education?
Alignment is more akin to indoctrination because education, in theory, makes you smarter and more open-minded.
OpenAI Reinforcement Fine-Tuning Research Program
41–50 of 67 posts
Re: OpenAI Reinforcement Fine-Tuning Research Program
#42Clever way to get more training data.
Can't you opt out? I'd even wager by default they don't retain this data for in-house training, especially at enterprise.
Re: OpenAI Reinforcement Fine-Tuning Research Program
#43What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
1) DPO did exclude some practical aspects of the RLHF method, e.g. pretraining gradients. 2) the theoretical arguments of DPO equivalence make some assumptions that don’t necessarily apply in practice 3) RLHF gives you a reusable reward model, which has practical uses and advantages. DPO doesn’t have useful intermediate product. 4) DPO works off preference, whereas desirable RL objectives could have many forms in pra…
[dead]
Re: OpenAI Reinforcement Fine-Tuning Research Program
#44Re: OpenAI Reinforcement Fine-Tuning Research Program
#45What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
This is not human feedback reinforcement learning, it is just traditional supervised reinforcement learning where the finetuning sets consist of problems and the correct answers. They do not call it supervised though because they have to say it is different than how they were finetuning until now.
Re: OpenAI Reinforcement Fine-Tuning Research Program
#46If I'd like to learn more about DPO and RLHF, I've been looking for toy problems/datasets to use but coming up a bit empty handed. Is there a convenient way to experiment with these methods through toy problems and simulation that can be done on a single GPU? The need for massive data and parameter counts to do anything interesting makes learning about these methods a little daunting.
Re: OpenAI Reinforcement Fine-Tuning Research Program
#47Re: OpenAI Reinforcement Fine-Tuning Research Program
#48If I'd like to learn more about DPO and RLHF, I've been looking for toy problems/datasets to use but coming up a bit empty handed. Is there a convenient way to experiment with these methods through toy problems and simulation that can be done on a single GPU? The need for massive data and parameter counts to do anything interesting makes learning about these methods a little daunting.
https://stable-baselines3.readthedocs.io/en/master/ is a great resource for hacking on implementations for RL - many good RL courses out there but https://www.youtube.com/playlist?list=PLwRJQ4m4UJjNymuBM9Rdm... is my personal favorite.
For LLMs / RLHF it's a little more difficult but https://github.com/huggingface/alignment-handbook and the Zephyr project is a good collection of model / dataset / script that is easy to follow.
I would suggest studying the basics of RL first before diving into LLM RLHF, which is much harder to learn on a single GPU.
Re: OpenAI Reinforcement Fine-Tuning Research Program
#49Earlier quoted context omitted.
Is it? We have no idea.
Yes it is. In RLHF and DPO you are optimizing the model output for human preferences. In the reinforcement fine tuning that was announced today you are optimizing the hidden chain of thought to arrive at a correct answer, as judged by a predefined grader.
I mean i think it could easily be PPO post training. if your point is that the rewards are different, sure
Re: OpenAI Reinforcement Fine-Tuning Research Program
#50What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
I think what people are missing here is that this is for o1, and you are supplying questions & answers, but not the entire solution-solving transcript (as you almost never have such a thing). The whole point of o1 is that you don't simply train on the supervised pairs that the users will be supplying here, because it's so hard to simply leap straight from a question to a correct answer, without doing additional work in between. (OA already offers a finetuning service like that, note.)
So DPO vs RLHF is missing the point: the interesting thing here is how they are (presumably) generating the inner-monologue to fill in the gap between the Q and the A that you provide them, and then training on that augmented dataset of Q->solving->A datapoints.
Whether they are using simple finetuning on that dataset, or DPO, or RLHF, or something else, seems less interesting than the broader questions of, "does that work? and are there many important or economically datasets where o1 can 'fill in the gaps', creating a better annotated dataset, and bootstrap itself to be much more intelligent on that dataset?"