Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The inefficiency of RL, and implications for RLVR progress

dwarkesh.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: The inefficiency of RL, and implications for RLVR progress

#41

Post by vessenes »

cheema33 wrote: post #38
andyjohnson0 wrote: post #2

Since it is not explicitly stated, "RL" in this article means Reinforcement Learning. https://en.wikipedia.org/wiki/Reinforcement_learning

Even though I knew which RL was being referred to here, the (ab)use of initials in this ways annoys me to no end. I wish people did not do that.

Counterpoint: much of academia is creating and learning these shorthands. They are genuinely useful - humans have limited context space in their heads, so this compression allows them to work in larger problem spaces. Classic example: Einstein and tensors.

Upshot - don’t hate - pick up the vocab, it’s part of the learning process.

Re: The inefficiency of RL, and implications for RLVR progress

#42

Post by yorwba »

hereme888 wrote: post #37
ACCount37 wrote: post #35

Earlier quoted context omitted.

Not necessarily. The reason why SFT can hurt performance is often the gap between the data and the capabilities. Imagine forcing someone who never used chopsticks to eat with the chopsticks. The results wouldn't be good - the instruction "use chopsticks" has taken effect, but an underlying "chopstick use" capability isn't there. If your SFT data pushes your LLM too far past its capabilities? It'll teach it to try doi…

fair point regarding data quality, but in the PEFT-Bench study, the base model actually outperformed the fine-tuned versions on those specific math/code tasks. So the "chopstick capability" was already there (at least partially), but the SFT process actively degraded it. It seems less about the data being too hard and more about the parameter-efficient methods (like LoRA) overwriting or interfering with delicate reas…

I think they must've messed up validation somehow. The performance drops relative to the base model are sometimes quite dramatic, which should've been caught by corresponding deterioration in validation performance.

They write "we utilize 10% randomly selected from the training set as a validation set and the original validation set as a test set for evaluation. During the validation phase, we measure validation loss and save the weights of the best validation loss for every 5% of the training steps. We train for 10 epochs with a batch size of 4." so it might be as simple as not including the base model in the validation checkpoints, meaning that the first validated checkpoint is after half an epoch, which is plenty of time to do damage if the fine-tuning method/hyperparameter configuration isn't chosen well. Unfortunately, they don't graph their training curves.

Re: The inefficiency of RL, and implications for RLVR progress

#43

Post by bugglebeetle »

vessenes wrote: post #40
bugglebeetle wrote: post #31

Earlier quoted context omitted.

Dwarkesh has a CS degree, but zero academic training or real world experience in deep learning, so all of his blogging is just secondhand bullshitting to further siphon off a veneer of expertise from his podcast guests.

So grumpy! Please pick up the torch and educate the world better; it can only help.

I teach and mentor lots of folks in my world. What I don’t do is feign expertise to rub shoulders with the people doing the actual work so I can soak money from rubes with ad rolls.

Re: The inefficiency of RL, and implications for RLVR progress

#44

Post by refulgentis »

vessenes wrote: post #40
bugglebeetle wrote: post #31

Earlier quoted context omitted.

Dwarkesh has a CS degree, but zero academic training or real world experience in deep learning, so all of his blogging is just secondhand bullshitting to further siphon off a veneer of expertise from his podcast guests.

So grumpy! Please pick up the torch and educate the world better; it can only help.

Better to be honest than say nothing, plenty of people say nothing. I asked a polite question thats near-impossible to answer without that level of honesty.

Re: The inefficiency of RL, and implications for RLVR progress

#45

Post by DevelopingElk »

ACCount37 wrote: post #33
bogtog wrote: post #10

The premise of this post and the one cited near the start ( https://www.tobyord.com/writing/inefficiency-of-reinforcemen... ) is that RL involves just 1 bit of learning for a rollout, rewarding success/failure. However, the way I'm seeing this is that a RL rollout may involve, say, 100 small decisions out of a pool of 1,000 possible decisions. Each training step, will slightly upregulate/downregulate a given training…

RL is very important - because while it's inefficient, and sucks at creating entirely new behaviors or features in LLMs, it excels at bringing existing features together and tuning them to perform well. It's a bit like LLM glue. The glue isn't the main material - but it's the one that holds it all together.

RL before LLMs can very much learn new behaviors. Take a look at AlphaGo for that. It can also learn to drive in simulated environments. RL in LLMs is not learning the same way, so it can't create it's own behaviors.

Re: The inefficiency of RL, and implications for RLVR progress

#46

Post by cubefox »

hereme888 wrote: post #26

recent results like PEFT-Bench (arxiv.org/abs/2511.21285) found that while SFT is efficient for formatting, it actually degraded Llama-3-8B's reasoning on math and code tasks compared to the base model. So is RL required to preserve those logic circuits? There seems to be a trade-off in compute-efficiency and format vs intelligence

With "supervised learning" he meant LLM pretraining, i.e., unsupervised / self-supervised learning. Not actual SFT.

Re: The inefficiency of RL, and implications for RLVR progress

#47

Post by vessenes »

refulgentis wrote: post #44
vessenes wrote: post #40

Earlier quoted context omitted.

So grumpy! Please pick up the torch and educate the world better; it can only help.

Better to be honest than say nothing, plenty of people say nothing. I asked a polite question thats near-impossible to answer without that level of honesty.

I thought your question was great. I read the Dwarkesh post as scratch space for working out his thinking - so, closer to a shower thought. But also, an attempt to do what he’s really great at, which is distill and summarize at a “random engineer” level of complexity.

You can kind of hear him pull in these extremely differing views on the future from very different sources, try and synthesize them, and also come out with some of his own perspective this year - I think it’s interesting. At the very least, his perspective is hyper-informed - he’s got fairly high-trust access to a lot of decision makers and senior researchers - and he’s smart and curious.

This year we’ve had him bring in the 2027 folks (AI explosion on schedule), Hinton (LLMs are literally divorced from reality, and a total dead-end), both Ilya (we probably need emotions for super intelligence, also I won’t tell you my plan), Karpathy and Dario (Dario maybe twice?), Gwen, all with very very different perspectives on what’s coming and why.

So, I think if you read him as one of the chroniclers of this era his own take is super interesting, and he’s in a position to be of great use precisely at synthesizing and (maybe) predicting; he should keep it up.

Re: The inefficiency of RL, and implications for RLVR progress

#49

Post by gpvos »

sidibe wrote: post #27
gpvos wrote: post #17

Earlier quoted context omitted.

[flagged]

Ok so now it's stupid or malicious to use RL as reinforcement learning on a blog about AI where everyone in the field has been referring to it as RL forever? Even wikipedia puts (RL) after reinforcement learning.

That's the normal way to introduce an acronym in an article.

Anyway, I was just saying that however irritating, it's likely just an omission out of forgetfulness, not deliberate clickbait. A minor application of Hanlon's razor.

Seeing the downvotes and even a flag, it appears I'll have to lower my expectation of people's cultural baggage here.

Re: The inefficiency of RL, and implications for RLVR progress

#50

Post by gpvos »

gpvos wrote: post #49
sidibe wrote: post #27

Earlier quoted context omitted.

Ok so now it's stupid or malicious to use RL as reinforcement learning on a blog about AI where everyone in the field has been referring to it as RL forever? Even wikipedia puts (RL) after reinforcement learning.

That's the normal way to introduce an acronym in an article. Anyway, I was just saying that however irritating, it's likely just an omission out of forgetfulness, not deliberate clickbait. A minor application of Hanlon's razor. Seeing the downvotes and even a flag, it appears I'll have to lower my expectation of people's cultural baggage here.

Additionally, replying to "in the field" in GP: this is about the article title. You first have to know which field the article is in, which simply is not clear if you are an HN reader that happens to not be in that field.
Post reply on HN