The inefficiency of RL, and implications for RLVR progress
1–10 of 50 posts
Re: The inefficiency of RL, and implications for RLVR progress
#2Since it is not explicitly stated, "RL" in this article means Reinforcement Learning.
Re: The inefficiency of RL, and implications for RLVR progress
#3In the limit, the "happy" case (positive reward), policy gradients boil down to performing more or less the same update as the usual supervised strategy for each generated token (or some subset of those if we use sampling). In the unhappy case, they penalise the model for selecting particular tokens in particular circumstances -- this is not something you can normally do with supervised learning, but it is unclear to what extent this is helpful (if a bad and a good answer share a prefix, it will be upvoted in one case and penalised in another case, not in the same exact way but still). So during on-policy learning we desperately need the model to stumble on correct answers often enough, and this can only happen if the model knows how to solve the problem to begin with, otherwise the search space is too big. In other words, while in supervised learning we moved away from providing models with inductive biases and trusting them to figure out everything by themselves, in RL this does not really seem possible.
Re: The inefficiency of RL, and implications for RLVR progress
#4Bit of a nitpick, but I think his terminology is wrong. Like RL, pretraining is also a form of *un*supervised learning
Re: The inefficiency of RL, and implications for RLVR progress
#5Bit of a nitpick, but I think his terminology is wrong. Like RL, pretraining is also a form of *un*supervised learning
Usual terminology for the three main learning paradigms:
- Supervised learning (e.g. matching labels to pictures)
- unsupervised learning / self-supervised learning (pretraining)
- reinforcement learning
Now the confusing thing is that Dwarkesh Patel instead calls pretraining "supervised learning" and you call reinforcement learning a form of unsupervised learning.
Re: The inefficiency of RL, and implications for RLVR progress
#6In the limit, the "happy" case (positive reward), policy gradients boil down to performing more or less the same update as the usual supervised strategy for each generated token (or some subset of those if we use sampling). In the unhappy case, they penalise the model for selecting particular tokens in particular circumstances -- this is not something you can normally do with supervised learning, but it is unclear to…
The trick is to provide dense rewards, i.e. not only once full goal is reached, but a little bit for every random flailing of the agent in the approximately correct direction.
Re: The inefficiency of RL, and implications for RLVR progress
#7Since it is not explicitly stated, "RL" in this article means Reinforcement Learning. https://en.wikipedia.org/wiki/Reinforcement_learning
I, too, started parsing this as RL=real life and that’s why I found the headline interesting
Re: The inefficiency of RL, and implications for RLVR progress
#8Bit of a nitpick, but I think his terminology is wrong. Like RL, pretraining is also a form of *un*supervised learning
Usual terminology for the three main learning paradigms: - Supervised learning (e.g. matching labels to pictures) - unsupervised learning / self-supervised learning (pretraining) - reinforcement learning Now the confusing thing is that Dwarkesh Patel instead calls pretraining "supervised learning" and you call reinforcement learning a form of unsupervised learning.
You could think of supervised learning as learning against a known ground truth, which pretraining certainly is.
Re: The inefficiency of RL, and implications for RLVR progress
#9In the limit, the "happy" case (positive reward), policy gradients boil down to performing more or less the same update as the usual supervised strategy for each generated token (or some subset of those if we use sampling). In the unhappy case, they penalise the model for selecting particular tokens in particular circumstances -- this is not something you can normally do with supervised learning, but it is unclear to…
The trick is to provide dense rewards, i.e. not only once full goal is reached, but a little bit for every random flailing of the agent in the approximately correct direction.
Article talks about all of this and references DeepSeek R1 paper[0], section 4.2 (first bullet point on PRM) on why this is much trickier to do than it appears.
Re: The inefficiency of RL, and implications for RLVR progress
#10The premise of this post and the one cited near the start (https://www.tobyord.com/writing/inefficiency-of-reinforcemen...) is that RL involves just 1 bit of learning for a rollout, rewarding success/failure.
However, the way I'm seeing this is that a RL rollout may involve, say, 100 small decisions out of a pool of 1,000 possible decisions. Each training step, will slightly upregulate/downregulate a given training step in the step's condition. There will be uncertainty about which decision was helpful/harmful -- we only have 1 bit of information after all -- but this setup where many steps are slowly learned across many examples seems like it would lend itself well to generalization (e.g., instead of 1 bit in one context, you get a hundred 0.01 bit insights across 100 contexts). There may be some benefits not captured by comparing the number of bits relative to pretraining.
As the blog says, "Fewer bits, sure, but very valuable bits", this also seems like a different factor that would also be true. Learning these small decisions may be vastly more valuable for producing accurate outputs than learning through pretraining.