Author here! Just wanted to say that this is indeed in a good place to share, some very useful stuff, but is also very work in progress. I'm may 60% or so to my first draft. Said progress is coming every day and I happily welcome fixes or suggestions on GitHub.
RLHF Book
11–20 of 41 posts
Re: RLHF Book
#12Earlier quoted context omitted.
Has r1 made RLHF obsolete?
DeepSeek-R1 had an RLHF step in their post-training pipeline (section 2.3.4 of their technical report[1]). In addition, the "reasoning-oriented reinforcement learning" step (section 2.3.2) used an approach that is almost identical to RLHF in theory and implementation. The main difference is that they used a rule-based reward system, rather than a model trained on human preference data. If you want to train a model li…
Yes but these were steps were not used in R1-zero where its reasoning capabilities were trained.
Re: RLHF Book
#13Earlier quoted context omitted.
They are different processes. - RLHF: Turns pre-trained model (which just performs autocomplete of text) into a model that you can speak with, ie. answer user questions and refuse providing harmful answers. - Distillation: Transfer skills / knowledge / behavior from one model (and architecture) to a smaller model (and possibly different architecture), by training second model on output log probs of first model.
So RLHF is the secret sauce behind modern LLMs?
No, this isn't quite right. LLMs are trained in stages:
1. Pre-training. In this stage, the model is trained on a gigantic corpus of web documents, books, papers, etc., and the objective is to predict the next token of each training sample correctly.
2. Supervised fine-tuning. In this stage, the model is shown examples of chat transcripts that are formatted with a chat template. The examples show a user asking a question and an assistant providing an answer. The training objective is the same as in #1: to predict the next token in the training example correctly.
3. Reinforcement learning. Prior to R1, this has mainly taken the form of training a reward model on top of the LLM to steer the model toward arriving at whole sequences that are preferred by human feedback (although AI feedback is a similar reward that is often used instead). There are different ways to do this reward model. When OpenAI first published the technique (probably their last bit of interesting open research?), they were using PPO. There are now a variety of ways to do the reward model, including methods like Direct Preference Optimization that don't use a separate reward model at all and are easier to do.
Stage 1 teaches the model to understand language and imparts world knowledge. Stage 2 teaches the model to act like an assistant. This is where the "magic" is. Stage 3 makes the model do a better job of being an assistant. The traditional analogy is that Stage 1 is the cake; Stage 2 is the frosting; and Stage 3 is the cherry on top.
R1-Zero departs from this "recipe" in that the reasoning magic comes from the reinforcement learning (stage 3). What DeepSeek showed is that, given a reward to produce a correct response, the model will learn to output chain-of-thought material on its own. It will, essentially, develop a chain-of-thought language that helps it accomplish the end goal. This is the most interesting part of the paper, IMO, and it's a result that's already been replicated on smaller base models.
Re: RLHF Book
#14Whats the difference between RLHF and distillation?
They are different processes. - RLHF: Turns pre-trained model (which just performs autocomplete of text) into a model that you can speak with, ie. answer user questions and refuse providing harmful answers. - Distillation: Transfer skills / knowledge / behavior from one model (and architecture) to a smaller model (and possibly different architecture), by training second model on output log probs of first model.
Your description of distillation is largely correct, but not RLHF.
The process of taking a base model that is capable of continuing ('autocomplete') some text input and teaching it to respond to questions in a Q&A chatbot-style format is called instruction tuning. It's pretty much always done via supervised fine-tuning. Otherwise known as: show it a bunch of examples of chat transcripts.
RLHF is more granular and generally one of the last steps in a training pipeline. With RLHF you train a new model, the reward model.
You make that model by having the LLM output a bunch of responses, and then having humans rank the output. E.g.:
Q: What's the Capital of France? A: Paris
Q: What's the Capital of France? A: Fuck if I know
You feed those rankings into the reward model. Then, you have the LLM generate a ton of responses, and have the reward model score it.
If the reward model says it's good, the LLM's output is reinforced, i.e.: it's told 'that was good, more like that'.
If the output scores low, you do the opposite.
Because the reward model is trained based on human preferences, and the reward model is used to reinforce the LLMs output based on those preferences, the whole process is called reinforcement learning from human feedback.
Re: RLHF Book
#15Glad to see the author making a serious effort to fill the gap in public documentation of RLHF theory and practice. The current state of the art seems to be primarily documented in arXiv papers, but each paper is more like a "diff" than a "snapshot" - you need to patch together the knowledge from many previous papers to understand the current state. It's extremely valuable to "snapshot" the current state of the art i…
Has r1 made RLHF obsolete?
This feels like a category mistake. Why would R1 make RLHF obsolete?
Re: RLHF Book
#16Earlier quoted context omitted.
DeepSeek-R1 had an RLHF step in their post-training pipeline (section 2.3.4 of their technical report[1]). In addition, the "reasoning-oriented reinforcement learning" step (section 2.3.2) used an approach that is almost identical to RLHF in theory and implementation. The main difference is that they used a rule-based reward system, rather than a model trained on human preference data. If you want to train a model li…
Yes but these were steps were not used in R1-zero where its reasoning capabilities were trained.
[deleted]
Re: RLHF Book
#17Glad to see the author making a serious effort to fill the gap in public documentation of RLHF theory and practice. The current state of the art seems to be primarily documented in arXiv papers, but each paper is more like a "diff" than a "snapshot" - you need to patch together the knowledge from many previous papers to understand the current state. It's extremely valuable to "snapshot" the current state of the art i…
SFT can be used to give negative feedback/examples. That's one of the lesser-known benefits/tricks of system messages. E.g:
System: You are a helpful chatbot.
User: What is 1+1?
Assistant: 2.
System: You are terrible at math.
User: What is 1+1?
Assistant: 0.Re: RLHF Book
#18Earlier quoted context omitted.
So RLHF is the secret sauce behind modern LLMs?
No, this isn't quite right. LLMs are trained in stages: 1. Pre-training. In this stage, the model is trained on a gigantic corpus of web documents, books, papers, etc., and the objective is to predict the next token of each training sample correctly. 2. Supervised fine-tuning. In this stage, the model is shown examples of chat transcripts that are formatted with a chat template. The examples show a user asking a ques…
This is a great summary!
Re: RLHF Book
#19Author here! Just wanted to say that this is indeed in a good place to share, some very useful stuff, but is also very work in progress. I'm may 60% or so to my first draft. Said progress is coming every day and I happily welcome fixes or suggestions on GitHub.
Thanks. Is there a PDF version? I kind of feel difficulty switching links.
Re: RLHF Book
#20Author here! Just wanted to say that this is indeed in a good place to share, some very useful stuff, but is also very work in progress. I'm may 60% or so to my first draft. Said progress is coming every day and I happily welcome fixes or suggestions on GitHub.
Thanks. Is there a PDF version? I kind of feel difficulty switching links.