11–20 of 67 posts
What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
In a final lecture at UC Berkeley this semester, Dawn Song was very clear that malicious fine tuning is a top priority among implementers right now. "Towards building safe and trustworthy AI Agents and a Path for Science- and Evidence-based AI Policy."
Say more…
Generalized finetunes meant to uncensor the model generally tend to underperform... but if you have a quality dataset for very specific task that typically would go against the alignment of the model, it's trivial to finetune on the task and get full performance down stream.
Who owns the fine tuning IP. Can OpenAI resell your model after investing a lot in it?
Of course, it isn't your IP free and clear either, because the base model isn't open so your fine-tuned model will always live inside OpenAI's walled garden.
If you're interested in reinforcement learning on top of truly open models where you own the end product, we're putting a lot of thought into that and are also looking for design partners! Feel free to email me at kyle@openpipe.ai.
What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
2) the theoretical arguments of DPO equivalence make some assumptions that don’t necessarily apply in practice
3) RLHF gives you a reusable reward model, which has practical uses and advantages. DPO doesn’t have useful intermediate product.
4) DPO works off preference, whereas desirable RL objectives could have many forms
in practice big labs are testing all these methods to see what works best.
Earlier quoted context omitted.
Say more…
You can strip most alignment from these models with finetuning. Generalized finetunes meant to uncensor the model generally tend to underperform... but if you have a quality dataset for very specific task that typically would go against the alignment of the model, it's trivial to finetune on the task and get full performance down stream.
There would appear to be a few issues with that, the most obvious being the uncensored model would presumably be "aligned" with what the finetuner wants.
Is there any piece I can read that gives an overview of the ways in which modern LLM networks are trained and optimized?
What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
Clever way to get more training data.