RL constrains the space of possible output token sequences to what is likely to lead to the correct answer. So we are inherently making a trade-off to reduce variance. A non-RL model will have higher variance, so given enough attempts, it will come up with some correct answers that an RL model can't.
Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
21–30 of 39 posts
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#22> we uncover that RL-trained models excel at low k (e.g., pass@1) but are consistently outperformed by base models at high k (e.g., pass@256). This is a weak argument. I think I get what we are trying to say, but let's take this to the extreme, say pass@10^10^100. Just like a group of monkeys could write Shakespeare if given enough time, a complete random model could probably outperform an RL-trained model at pass@10…
If this was just the effect you mention you would not expect the base model to surpass the RL model though. Plus their k are much smaller than that.
I think it's a very interesting and meaningful study.
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#23Earlier quoted context omitted.
At least with Deep Seek math (with the same RL technique as the later R1) they noted similar things in their paper in the "Why RL Works?" section. Around the 1:04:00 mark of this Yannic Kilcher video review of the Deepseek math paper he goes over that section and points to basically the same limitations as the hn submission paper, starts at around the 1hr 4m mark and ends with this: 1:05:40 the Improvement is attribu…
I don't know a lot about this but it seems like if the sampling performance was adequate, external checks like theorem verification would work to get "over the data wall."
There have already been good results there with DeepMind's math Olympiad work. I think the LLM portion there was only for translating from informal to formal in the training process and in the final process they still used a manual translation to a formal description and the solver was transformer based and RL trained, but I think not starting with any language base, but it was able to learn some distribution helpful in solving the problems with RL, verifier,and light scaffolding of the tree search alone.
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#24Earlier quoted context omitted.
That depends a bit on the length of the RL training and the distribution of problems you're training on. You're correct that RL won't get any "traction" (via positive rewards) on problems where good behavior isn't already in the model's behavior distribution. However, if you're training on many problems, it's possible in principle that if you have traction on _any_ of the problems, then the learning signal you get fr…
This is an interesting scenario: do you know of any documented examples?
Offhand, I don't know any specific examples for LLMs. In general though, if you google something like "automated curriculum design for reinforcement learning", you should find some relevant references.
Some straightforward scenarios are in, eg, robotics where one can design sequences of increasingly difficult instances of a task like moving objects from one storage bin to another. The basic idea is that the agent would have no reward or learning signal if it jumped straight into the full version of the task, so you let it develop competence on simpler variants and gradually increase difficulty until the agent can get useful learning signal on the full task.
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#25Earlier quoted context omitted.
Hallucinations usually happen when a model never knew the answer, not when it forgot something.
I think this is definitely not true of catastrophic forgetting from finetuning. And with other related types of forgetting from model abliteration there are often extreme increases hallucination. The InstructGPT paper also showed that RLHF made hallucination worse (with more user data rejecting common hallucinations instruction tuning and RLHF may lower specific hallucinations rejected by users though). Some mention…
RL might be making hallucinations worse, that’s true. Why do you think RL is causing catastrophic forgetting? Are there factual knowledge benchmarks showing it for o3 or o4-mini?
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#26I don't like papers that ask a question in the title, so here's the answer: "RL boosts sampling efficiency but reduces the reasoning capacity boundary." Perhaps better to put it like this: Given one, or few attempts, RL trained models beat non-RL models. Given many attempts, non-RL models come up with better answers.
I'm pretty sure RL causes catastrophic forgetting of its base knowledge and that's why o3 hallucinates so much more. If you mess around with trained weights you're going to delete some base knowledge, as least the knowledge that is outside of the tasks you RL on.
Solution could be to mix RL training with foundational knowledge training, so LLM can refresh memory and not forget things.
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#27> we uncover that RL-trained models excel at low k (e.g., pass@1) but are consistently outperformed by base models at high k (e.g., pass@256). This is a weak argument. I think I get what we are trying to say, but let's take this to the extreme, say pass@10^10^100. Just like a group of monkeys could write Shakespeare if given enough time, a complete random model could probably outperform an RL-trained model at pass@10…
The authors of the paper address this argument in the QA section.
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#28Earlier quoted context omitted.
I think this is definitely not true of catastrophic forgetting from finetuning. And with other related types of forgetting from model abliteration there are often extreme increases hallucination. The InstructGPT paper also showed that RLHF made hallucination worse (with more user data rejecting common hallucinations instruction tuning and RLHF may lower specific hallucinations rejected by users though). Some mention…
RL might be making hallucinations worse, that’s true. Why do you think RL is causing catastrophic forgetting? Are there factual knowledge benchmarks showing it for o3 or o4-mini?
Just since any continued training tends to cause catastrophic forgetting if the old info isn't regurgitated again.
Not specifically showing catastrophic forgetting, but hallucination for o3:
> From the results of this evaluation, o3's hallucination rate is 33 percent, and o4-mini's hallucination rate is 48 percent — almost half of the time. By comparison, o1's hallucination rate is 16 percent, meaning o3 hallucinated about twice as often.
Deepseek R1 handles some of this by redistilling back in "factual Q&A" generated from original V3 model to make a new V3. The V3 paper mentions it incorporated an R1 pass too so it seems like: V3 base model, RL pass, V3 with RL distill and retraining a checkpoint for the final V3 release, additional RL pass for the final R1 release.
V3 Paper
> During the post-training stage, we distill the reasoning capability from the DeepSeekR1 series of models [I think that refers to the earlier checkpoint R1 after the first pass below]
R1 Paper:
> To address these issues and further enhance reasoning performance, we introduce DeepSeek-R1, which incorporates a small amount of cold-start data and a multi-stage training pipeline. Specifically, we begin by collecting thousands of cold-start data to fine-tune the DeepSeek-V3-Base model. Following this, we perform reasoning-oriented RL like DeepSeek-R1-Zero. Upon nearing convergence in the RL process, we create new SFT data through rejection sampling on the RL checkpoint, combined with supervised data from DeepSeek-V3 in domains such as writing, factual QA, and self-cognition, and then retrain the DeepSeek-V3-Base model. After fine-tuning with the new data, the checkpoint undergoes an additional RL process, taking into account prompts from all scenarios. After these steps, we obtained a checkpoint referred to as DeepSeek-R1, which achieves performance on par with OpenAI-o1-1217.
In general with fine-tuning you can avoid catastrophic forgetting by mixing in the original data during later fine tuning steps, and from this it seems the same is true of the RL phases, but they are also doing some amount of augmentation and selection on the the data involved.
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#29Reasoning models aren't really reasoners, its basically neural style transfer protocol where you force a model "decoder" to emit tokens in a style that appears to be Reasoning like a deductive thinking.
Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?
#30Earlier quoted context omitted.
At least with Deep Seek math (with the same RL technique as the later R1) they noted similar things in their paper in the "Why RL Works?" section. Around the 1:04:00 mark of this Yannic Kilcher video review of the Deepseek math paper he goes over that section and points to basically the same limitations as the hn submission paper, starts at around the 1hr 4m mark and ends with this: 1:05:40 the Improvement is attribu…
Thanks for sharing. I had trouble reading the transcript, so here is Claude's cleaned up version and summary: Here's the condensed and formatted transcription in a single paragraph: This is the last thing I want to highlight this section on why RL works. Here they evaluate different things - they evaluate specifically pass at K and maj at K. Maj at K is like majority voting, so what you do is you have a model, you ha…
This also seems to be why rejection sampling + SFT seems just as good if not better in a lot of scenarios