31–40 of 41 posts
A quote I found helpful: "reinforcement learning from human feedback .. is designed to optimize machine learning models in domains where specifically designing a reward function is hard" https://rlhfbook.com/c/05-preferences.html
Are there any books (on RL) which are more hands-on and look at the implementations more than the theory?
Whats the difference between RLHF and distillation?
They are different processes. - RLHF: Turns pre-trained model (which just performs autocomplete of text) into a model that you can speak with, ie. answer user questions and refuse providing harmful answers. - Distillation: Transfer skills / knowledge / behavior from one model (and architecture) to a smaller model (and possibly different architecture), by training second model on output log probs of first model.
Earlier quoted context omitted.
They are different processes. - RLHF: Turns pre-trained model (which just performs autocomplete of text) into a model that you can speak with, ie. answer user questions and refuse providing harmful answers. - Distillation: Transfer skills / knowledge / behavior from one model (and architecture) to a smaller model (and possibly different architecture), by training second model on output log probs of first model.
Your description of distillation is largely correct, but not RLHF. The process of taking a base model that is capable of continuing ('autocomplete') some text input and teaching it to respond to questions in a Q&A chatbot-style format is called instruction tuning. It's pretty much always done via supervised fine-tuning. Otherwise known as: show it a bunch of examples of chat transcripts. RLHF is more granular and gen…
Here is presentation by Karpathy explaining different stages of LLM training. Explains many details in a form suitable for beginners.
A quote I found helpful: "reinforcement learning from human feedback .. is designed to optimize machine learning models in domains where specifically designing a reward function is hard" https://rlhfbook.com/c/05-preferences.html
How do we draw the line between a hard and not-so-hard reward function?
For example, say you’re building a chess AI that you’re going to train using reinforcement learning alphazero-style. No matter how fancy the logic that you want to employ to build the AI itself, it’s really easy to make a reward function. “Did it win the game” is the reward function.
On the other hand, if you’re making an AI to write poetry. It’s hard/impossible to come up with an objective function to judge the output so you use RLHF.
It lots of cases the whole design springs from the fact that it’s hard to make a suitable reward function (eg GANs for generation of realistic faces is the classic example). What makes an image of a face realistic? So Goodfellow came up with the idea of having two nets one which tries to generate and one which tries to discern which images are fake and which real. Now the reward functions are easy. The generator gets rewarded for generating images good enough to fool the classifier and the classifier gets rewarded for being able to spot which images are fake and which real.
Need epub version of this