Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The inefficiency of RL, and implications for RLVR progress

dwarkesh.com

Post reply on HN
Print view View original

31–40 of 50 posts

Re: The inefficiency of RL, and implications for RLVR progress

#31

Post by bugglebeetle »

refulgentis wrote: post #23
bogtog wrote: post #10

The premise of this post and the one cited near the start ( https://www.tobyord.com/writing/inefficiency-of-reinforcemen... ) is that RL involves just 1 bit of learning for a rollout, rewarding success/failure. However, the way I'm seeing this is that a RL rollout may involve, say, 100 small decisions out of a pool of 1,000 possible decisions. Each training step, will slightly upregulate/downregulate a given training…

Dwarkesh's blogging confuses me, because I am not sure if the message is free-associating, or, relaying information gathered. ex. how this reads if it is free-associating: "shower thought: RL on LLMs is kinda just 'did it work or not?' and the answer is just 'yes or no', yes or no is a boolean, a boolean is 1 bit, then bring in information theory interpretation of that, therefore RL doesn't give nearly as much info a…

Dwarkesh has a CS degree, but zero academic training or real world experience in deep learning, so all of his blogging is just secondhand bullshitting to further siphon off a veneer of expertise from his podcast guests.

Re: The inefficiency of RL, and implications for RLVR progress

#32

Post by robotresearcher »

jsnell wrote: post #20
Jaxan wrote: post #13

Earlier quoted context omitted.

How do you know the correct direction? Isn’t the point of learning that the right path is unknown to start with?

The correct solutions and the viable paths probably are known to the trainers, just not to the trainee. Training only on problems where the solution is unknown but verifiable sounds like the ultimate hard mode, and pretty hard to justify unless you have a model that's already saturated the space of problems with known solutions. (Actually, "pretty hard to justify" might be understating it. How can we confidently extr…

Your hard mode is exactly the situation that RL is used, because it requires neither a corpus of correct examples, nor insight into the structure of a good policy.

> How can we confidently extract any signal from a failure to solve a problem if we don't even know if the problem is solvable?)

You rule out all the stuff that doesn’t work.

Yes this is difficult and usually very costly. Credit assignment is a deep problem. But if you didn’t find yourself in a hard mode situation, you wouldn’t be using RL.

Re: The inefficiency of RL, and implications for RLVR progress

#33

Post by ACCount37 »

bogtog wrote: post #10

The premise of this post and the one cited near the start ( https://www.tobyord.com/writing/inefficiency-of-reinforcemen... ) is that RL involves just 1 bit of learning for a rollout, rewarding success/failure. However, the way I'm seeing this is that a RL rollout may involve, say, 100 small decisions out of a pool of 1,000 possible decisions. Each training step, will slightly upregulate/downregulate a given training…

RL is very important - because while it's inefficient, and sucks at creating entirely new behaviors or features in LLMs, it excels at bringing existing features together and tuning them to perform well.

It's a bit like LLM glue. The glue isn't the main material - but it's the one that holds it all together.

Re: The inefficiency of RL, and implications for RLVR progress

#34

Post by dang »

on_the_train wrote: post #16
Angostura wrote: post #11

Earlier quoted context omitted.

Thank god. Was driving me mad.

[flagged]

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

https://news.ycombinator.com/newsguidelines.html

Re: The inefficiency of RL, and implications for RLVR progress

#35

Post by ACCount37 »

hereme888 wrote: post #26

recent results like PEFT-Bench (arxiv.org/abs/2511.21285) found that while SFT is efficient for formatting, it actually degraded Llama-3-8B's reasoning on math and code tasks compared to the base model. So is RL required to preserve those logic circuits? There seems to be a trade-off in compute-efficiency and format vs intelligence

Not necessarily. The reason why SFT can hurt performance is often the gap between the data and the capabilities.

Imagine forcing someone who never used chopsticks to eat with the chopsticks. The results wouldn't be good - the instruction "use chopsticks" has taken effect, but an underlying "chopstick use" capability isn't there.

If your SFT data pushes your LLM too far past its capabilities? It'll teach it to try doing a thing it can't do.

If your SFT traces assume your LLM can do 10 digit multiplication, the LLM wouldn't learn 10 digit multiplication from them. It'll learn to attempt 10 digit multiplication, and it'll fail.

Re: The inefficiency of RL, and implications for RLVR progress

#36

Post by on_the_train »

dang wrote: post #34
on_the_train wrote: post #16

Earlier quoted context omitted.

[flagged]

" Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith. " https://news.ycombinator.com/newsguidelines.html

That doesn't, or shouldn't apply to the content itself. Because we all know how prevalent clickbait is.

Re: The inefficiency of RL, and implications for RLVR progress

#37

Post by hereme888 »

ACCount37 wrote: post #35
hereme888 wrote: post #26

recent results like PEFT-Bench (arxiv.org/abs/2511.21285) found that while SFT is efficient for formatting, it actually degraded Llama-3-8B's reasoning on math and code tasks compared to the base model. So is RL required to preserve those logic circuits? There seems to be a trade-off in compute-efficiency and format vs intelligence

Not necessarily. The reason why SFT can hurt performance is often the gap between the data and the capabilities. Imagine forcing someone who never used chopsticks to eat with the chopsticks. The results wouldn't be good - the instruction "use chopsticks" has taken effect, but an underlying "chopstick use" capability isn't there. If your SFT data pushes your LLM too far past its capabilities? It'll teach it to try doi…

fair point regarding data quality, but in the PEFT-Bench study, the base model actually outperformed the fine-tuned versions on those specific math/code tasks.

So the "chopstick capability" was already there (at least partially), but the SFT process actively degraded it. It seems less about the data being too hard and more about the parameter-efficient methods (like LoRA) overwriting or interfering with delicate reasoning circuits just to satisfy the formatting loss.

Re: The inefficiency of RL, and implications for RLVR progress

#38

Post by cheema33 »

andyjohnson0 wrote: post #2

Since it is not explicitly stated, "RL" in this article means Reinforcement Learning. https://en.wikipedia.org/wiki/Reinforcement_learning

Even though I knew which RL was being referred to here, the (ab)use of initials in this ways annoys me to no end. I wish people did not do that.

Re: The inefficiency of RL, and implications for RLVR progress

#39

Post by dash2 »

derbOac wrote: post #22

There's some insights there about the base rate of correct responses and pretraining to boost that. Basically searching a suboptimal versus optimal area of the model space at a suboptimal versus optimal rate. I think the framing of the discussion in general is kind of misleading though, because it kind of avoids the question of "information inefficient about what?" In RL, the model is becoming more informative about…

SL = supervised learning, right?

Re: The inefficiency of RL, and implications for RLVR progress

#40

Post by vessenes »

bugglebeetle wrote: post #31
refulgentis wrote: post #23

Earlier quoted context omitted.

Dwarkesh's blogging confuses me, because I am not sure if the message is free-associating, or, relaying information gathered. ex. how this reads if it is free-associating: "shower thought: RL on LLMs is kinda just 'did it work or not?' and the answer is just 'yes or no', yes or no is a boolean, a boolean is 1 bit, then bring in information theory interpretation of that, therefore RL doesn't give nearly as much info a…

Dwarkesh has a CS degree, but zero academic training or real world experience in deep learning, so all of his blogging is just secondhand bullshitting to further siphon off a veneer of expertise from his podcast guests.

So grumpy! Please pick up the torch and educate the world better; it can only help.
Post reply on HN