This was announced as part of their second day of "12 Days of AI": https://www.youtube.com/watch?v=fMJMhBFa_Gc
They're searching for enterprise customers before they become a commodity.
This was obvious even before the Microsoft deal got penned.
OpenAI Reinforcement Fine-Tuning Research Program
31–40 of 67 posts
Re: OpenAI Reinforcement Fine-Tuning Research Program
#32Re: OpenAI Reinforcement Fine-Tuning Research Program
#33Re: OpenAI Reinforcement Fine-Tuning Research Program
#34What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
On the topic of DPO - I have a Colab notebook to finetune with Unsloth 2x faster and use 50% less memory for DPO if it helps anyone! https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-h...
thank you !
Re: OpenAI Reinforcement Fine-Tuning Research Program
#35What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
In short, DPO is not better than PPO. This is because DPO is derived from so called BT reward assumption that pairwise data preference is collected. Through mathematical formulations, you can learn the preference and the action at the same time. However, PPO and other on-policy (training samples are strictly generated by the LLM) doesn't need such assumption. For example, in coding and math problems it is possible to get binary reward. Many research shows DPO is ok if you don't take much care on OOD performance.
Re: OpenAI Reinforcement Fine-Tuning Research Program
#36For security & fraud teams who want to 'own their AI' vs trust with Sam Altman, we are doing some fun things here as part of louie.ai, and looking for our next cohort of Splunk/databricks/elastic/neo4j/etc teams. LMK or signup on louie.ai -- I do agree with the direction openai is going, but as always, devil is in the details, and especially for serious problems on sensitive data.
Re: OpenAI Reinforcement Fine-Tuning Research Program
#37Earlier quoted context omitted.
Note that this reinforcement finetuning is something different than regular RLHF/DPO post training
Is it? We have no idea.
Yes it is. In RLHF and DPO you are optimizing the model output for human preferences. In the reinforcement fine tuning that was announced today you are optimizing the hidden chain of thought to arrive at a correct answer, as judged by a predefined grader.
Re: OpenAI Reinforcement Fine-Tuning Research Program
#38Re: OpenAI Reinforcement Fine-Tuning Research Program
#39Are alignment and fine-tuning just a parallel of education?
Re: OpenAI Reinforcement Fine-Tuning Research Program
#40this sounds like expert systems 2.0 lol
I assume it's more like scaled nlp, which sort of describes the whole thing to begin with. i suspect it will boil down to further generalizing nlp-in-the-loop algorithms, more Tools, Tools between Tools, presumably Expert mixtures or randomly selecting "Axioms" and having an expert forget one and seeing if what remains makes sense still as the Tools are operated, and how that can be encoded better across domains.
It's not nothing, but there's a lot of value stuck up in there, I mean, it's made out of people.
Real special, takes a lot of smart