Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Reinforcement Fine-Tuning Research Program

openai.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: OpenAI Reinforcement Fine-Tuning Research Program

#51

Post by eggie5 »

whimsicalism wrote: post #21
brandonb wrote: post #5

What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.

Most of the other replies to you, except for the one by tempusalaria, are not really answering the question. Broadly, while there was a lot of initial excitement - it simply does not seem like offline + off-policy RL can beat online + on-policy RL methods like PPO. Sampling trajectories from the actual model you are training and scoring them seems like it works much better in practice, never mind the additional flexi…

What's _online_ RL for an LLM? Saw this on the llama 3.3 reports too...

Re: OpenAI Reinforcement Fine-Tuning Research Program

#52

Post by whimsicalism »

eggie5 wrote: post #51
whimsicalism wrote: post #21

Earlier quoted context omitted.

Most of the other replies to you, except for the one by tempusalaria, are not really answering the question. Broadly, while there was a lot of initial excitement - it simply does not seem like offline + off-policy RL can beat online + on-policy RL methods like PPO. Sampling trajectories from the actual model you are training and scoring them seems like it works much better in practice, never mind the additional flexi…

What's _online_ RL for an LLM? Saw this on the llama 3.3 reports too...

Online RL for LLMs means you are sampling from the model, scoring immediately, and passing gradients back to the model.

As opposed to, sampling from the model a bunch, getting scores offline, and then fine tuning the model on those offline scored generations.

Re: OpenAI Reinforcement Fine-Tuning Research Program

#53

Post by griomnib »

echelon wrote: post #28
throwup238 wrote: post #6

This was announced as part of their second day of "12 Days of AI": https://www.youtube.com/watch?v=fMJMhBFa_Gc

They're searching for enterprise customers before they become a commodity.

Llama 3.3 is insanely good and can be run on a Mac mini with 64GB of ram for $2k USD.

OpenAI is screwed.

(As an aside: very interesting Google tried to go closed source and objectively lost the race, and Meta went open and is the real threat to OpenAI.)

Re: OpenAI Reinforcement Fine-Tuning Research Program

#54

Post by AdieuToLogic »

griomnib wrote: post #53
echelon wrote: post #28

Earlier quoted context omitted.

They're searching for enterprise customers before they become a commodity.

Llama 3.3 is insanely good and can be run on a Mac mini with 64GB of ram for $2k USD. OpenAI is screwed. (As an aside: very interesting Google tried to go closed source and objectively lost the race, and Meta went open and is the real threat to OpenAI.)

> OpenAI is screwed.

They are for multiple reasons, not the least of which is:

https://www.wheresyoured.at/subprimeai/

Re: OpenAI Reinforcement Fine-Tuning Research Program

#56

Post by radarsat1 »

roborovskis wrote: post #48
radarsat1 wrote: post #46

If I'd like to learn more about DPO and RLHF, I've been looking for toy problems/datasets to use but coming up a bit empty handed. Is there a convenient way to experiment with these methods through toy problems and simulation that can be done on a single GPU? The need for massive data and parameter counts to do anything interesting makes learning about these methods a little daunting.

https://stable-baselines3.readthedocs.io/en/master/ is a great resource for hacking on implementations for RL - many good RL courses out there but https://www.youtube.com/playlist?list=PLwRJQ4m4UJjNymuBM9Rdm... is my personal favorite. For LLMs / RLHF it's a little more difficult but https://github.com/huggingface/alignment-handbook and the Zephyr project is a good collection of model / dataset / script that is easy…

Hi, the Zephyr link may be what I'm looking for. yeah I'm quite familiar with RL already so it was specifically RLHF that I was asking about, I'll check out that resource, thanks!

Re: OpenAI Reinforcement Fine-Tuning Research Program

#57

Post by esperent »

kcorbitt wrote: post #13
patrickhogan1 wrote: post #4

Who owns the fine tuning IP. Can OpenAI resell your model after investing a lot in it?

No, generally speaking OpenAI doesn't re-use training data between customers. It's worth it to them anyway because they learn what does/doesn't work on different tasks Of course, it isn't your IP free and clear either, because the base model isn't open so your fine-tuned model will always live inside OpenAI's walled garden. If you're interested in reinforcement learning on top of truly open models where you own the e…

> No, generally speaking OpenAI doesn't re-use training data between customers

How do you know this?

Re: OpenAI Reinforcement Fine-Tuning Research Program

#58

Post by Tepix »

griomnib wrote: post #53
echelon wrote: post #28

Earlier quoted context omitted.

They're searching for enterprise customers before they become a commodity.

Llama 3.3 is insanely good and can be run on a Mac mini with 64GB of ram for $2k USD. OpenAI is screwed. (As an aside: very interesting Google tried to go closed source and objectively lost the race, and Meta went open and is the real threat to OpenAI.)

With 64GB you only get a lower quality quantized version.

Re: OpenAI Reinforcement Fine-Tuning Research Program

#59

Post by griomnib »

Tepix wrote: post #58
griomnib wrote: post #53

Earlier quoted context omitted.

Llama 3.3 is insanely good and can be run on a Mac mini with 64GB of ram for $2k USD. OpenAI is screwed. (As an aside: very interesting Google tried to go closed source and objectively lost the race, and Meta went open and is the real threat to OpenAI.)

With 64GB you only get a lower quality quantized version.

That’s the one I’m using. So far it’s quite good, and when I gave it and Claude the same programming problem not only did Llama give a better result, when I showed that result to Claude it also said the Llama approach was better.

Claude is already better than GPT on average at coding, so yeah, bad news for OpenAI as Llama is now potentially better at coding.

Of course Meta has a propriety training set of extremely high quality code, so if they are using that, I’d expect them to have vastly superior performance as FAANG production code is better training data than dogshit stack overflow questions to CS homework problems.

I really think whatever boost OpenAI get from their shadow CoT loop is nominal at best, but with 2x+ the amount of compute forcing them to increase prices an absurd amount.

It’s business 101, they just won’t make the revenue to cover those extra tokens and they are now competing against free. The economics do not suggest OpenAI has a path to survival without major breakthroughs in performance AND efficiency.

Re: OpenAI Reinforcement Fine-Tuning Research Program

#60

Post by whydid »

brandonb wrote: post #5

What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.

Recording good audio remains more difficult than artificial intelligence.
Post reply on HN