Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

limit-of-rlvr.github.io

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#31

Post by mountainriver »

cma wrote: post #28
kadushka wrote: post #25

Earlier quoted context omitted.

RL might be making hallucinations worse, that’s true. Why do you think RL is causing catastrophic forgetting? Are there factual knowledge benchmarks showing it for o3 or o4-mini?

Just since any continued training tends to cause catastrophic forgetting if the old info isn't regurgitated again. Not specifically showing catastrophic forgetting, but hallucination for o3: > From the results of this evaluation, o3's hallucination rate is 33 percent, and o4-mini's hallucination rate is 48 percent — almost half of the time. By comparison, o1's hallucination rate is 16 percent, meaning o3 hallucinated…

There has always been a post training phase with RLHF though since GPT 3.5

It’s nothing new, and it’s worked great for a long time. The difference now is RLVR, which yes I do suspect is causing it to over optimize to verifiable tasks and is probably losing a lot of nuanced information

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#32

Post by mountainriver »

macleginn wrote: post #5

‘Crucially, all correct solutions from RL-trained models already exist in the base model's distribution, proving RLVR enhances sampling efficiency, not reasoning capacity, while inadvertently shrinking the solution space.’ — wouldn't any kind of RL fail to converge or even progress at all if the solution weren't to be found in the base model distribution? The way training is set up, the models absolutely need to be a…

I felt like this was already known right? My understanding was always that the base model had all the paths and RL was learning to navigate them

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#33

Post by kadushka »

cma wrote: post #28
kadushka wrote: post #25

Earlier quoted context omitted.

RL might be making hallucinations worse, that’s true. Why do you think RL is causing catastrophic forgetting? Are there factual knowledge benchmarks showing it for o3 or o4-mini?

Just since any continued training tends to cause catastrophic forgetting if the old info isn't regurgitated again. Not specifically showing catastrophic forgetting, but hallucination for o3: > From the results of this evaluation, o3's hallucination rate is 33 percent, and o4-mini's hallucination rate is 48 percent — almost half of the time. By comparison, o1's hallucination rate is 16 percent, meaning o3 hallucinated…

Catastrophic forgetting would degrade factual knowledge benchmark results - even more than hallucination benchmarks, right? Do we observe this with o3/o4-mini? If not, your hypothesis is invalidated.

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#34

Post by cma »

kadushka wrote: post #33
cma wrote: post #28

Earlier quoted context omitted.

Just since any continued training tends to cause catastrophic forgetting if the old info isn't regurgitated again. Not specifically showing catastrophic forgetting, but hallucination for o3: > From the results of this evaluation, o3's hallucination rate is 33 percent, and o4-mini's hallucination rate is 48 percent — almost half of the time. By comparison, o1's hallucination rate is 16 percent, meaning o3 hallucinated…

Catastrophic forgetting would degrade factual knowledge benchmark results - even more than hallucination benchmarks, right? Do we observe this with o3/o4-mini? If not, your hypothesis is invalidated.

Hallucinations heavily are correlated with poor performance on factual benchmarks, or at least I was referring to hallucinating false facts it was never trained on.

There's a subset of hallucinations though where it hallucinates real factual things that weren't in the context as if they were there, but I think reasoning models improve on those since they deal with much longer strings of thought than typical internet fare, but maybe that's wrong. Deepseek reported heavily improved long context benchmark performance in r1 vs v3.

You could characterize "did needle occur in haystack of text? response: yes" as a hallucination, but those weren't what I was referring to. But they do seem to improve on those types of tasks after RL and reduce that kind of hallucination.

If it knows well what it doesn't know it could do well on a hallucination benchmark, while still doing worse on a factual breadth benchmark, but catastrophic forgetting degrades many other abilities and not just knowledge so I would tend to think it would degrade that too. At some point claude got much better about knowing what it doesn't know, I don't know if that was an emergent or trained ability, or if they did something more hand made like giving it access to logits of tokens or of contextual embeddings of what it previously generated.

Edit: just guessed on that last one but apparently there is a paper that tried that: https://arxiv.org/html/2409.06601v1

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#35

Post by zaptrem »

cma wrote: post #6
spwa4 wrote: post #3

I don't like papers that ask a question in the title, so here's the answer: "RL boosts sampling efficiency but reduces the reasoning capacity boundary." Perhaps better to put it like this: Given one, or few attempts, RL trained models beat non-RL models. Given many attempts, non-RL models come up with better answers.

I'm pretty sure RL causes catastrophic forgetting of its base knowledge and that's why o3 hallucinates so much more. If you mess around with trained weights you're going to delete some base knowledge, as least the knowledge that is outside of the tasks you RL on.

I think when they were figuring out RLHF they avoided this by interleaving RLHF and normal cross entropy on training set gradients.

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#36

Post by ismepornnahi »

imtringued wrote: post #12

>Our key finding is that all reasoning paths in the RLVR model are already present in the base model. This is a really good observation. It means that you don't need to RL the full model. You merely need to RL a few LoRAs or maybe a small Mamba model appended to the final layer.

Interesting, has this already been experimented?

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#37

Post by spwa4 »

GloamingNiblets wrote: post #18
cma wrote: post #16

Earlier quoted context omitted.

At least with Deep Seek math (with the same RL technique as the later R1) they noted similar things in their paper in the "Why RL Works?" section. Around the 1:04:00 mark of this Yannic Kilcher video review of the Deepseek math paper he goes over that section and points to basically the same limitations as the hn submission paper, starts at around the 1hr 4m mark and ends with this: 1:05:40 the Improvement is attribu…

Thanks for sharing. I had trouble reading the transcript, so here is Claude's cleaned up version and summary: Here's the condensed and formatted transcription in a single paragraph: This is the last thing I want to highlight this section on why RL works. Here they evaluate different things - they evaluate specifically pass at K and maj at K. Maj at K is like majority voting, so what you do is you have a model, you ha…

Just don't.

This is a horrible summary. It is both too complex and to simple at the same time. This summary spends about half it's time talking about pass@k while failing to explain what it is and giving a great deal of good-sounding but misleading statements, making me think Claude completely misunderstood (it is absolutely not like majority voting). Pass@k means you get k attempts to answer a question. Right? You passed. Wrong? Well, you've got k (for example 10) attempts.

The paper itself is much better. Hell, the conclusion of the paper is so much better than what you have here.

Here's a decent summary, directly from the paper's conclusion:

1. RL-trained models perform worse than base models in pass@k at large k values. (note that Claude's explanation of what pass@k is in the parent post is extremely wrong)

2. RL boosts sampling efficiency but reduces the reasoning capacity boundary.

3. RLVR algorithms perform similarly and remain far from optimal.

4. RLVR and distillation are fundamentally different.

And here's a one-line summary from me:

This paper claims that RL(VR) training is like improving the model's search algorithm: it becomes (a lot) better at locating a good answer within the model, but also pushes the model too hard to give only this answer.

Before Claude makes another absurd claim RL = reinforcement learning (for example, for safety. Say, trying to get the model to explain breaking into a car, if it ever does, that's bad), RLVR = reinforcement learning with verifiable rewards (meaning you get to think as much as you want, as long as your final answer is correct. But you get to reminisce/think as much as you want before giving a final answer, and that thinking does not have to be relevant)

And a comment: this is exactly what you'd expect to see from mild overtraining of the model. It could be that the current big players are pushing the models to be right/helpful/safe too hard, and taking away too much "freedom" in the process.

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#38

Post by GloamingNiblets »

spwa4 wrote: post #37
GloamingNiblets wrote: post #18

Earlier quoted context omitted.

Thanks for sharing. I had trouble reading the transcript, so here is Claude's cleaned up version and summary: Here's the condensed and formatted transcription in a single paragraph: This is the last thing I want to highlight this section on why RL works. Here they evaluate different things - they evaluate specifically pass at K and maj at K. Maj at K is like majority voting, so what you do is you have a model, you ha…

Just don't. This is a horrible summary. It is both too complex and to simple at the same time. This summary spends about half it's time talking about pass@k while failing to explain what it is and giving a great deal of good-sounding but misleading statements, making me think Claude completely misunderstood (it is absolutely not like majority voting). Pass@k means you get k attempts to answer a question. Right? You p…

I appreciate the feedback, another reminder to not lean too much on LLMs.

Re: Does RL Incentivize Reasoning in LLMs Beyond the Base Model?

#39

Post by imenani »

They fix the temperature at T=0.6 for all k for all models, even though their own Figure 10 shows that RL model benefits from higher temperatures. I would buy the overall claim much more if they swept of temperature parameter for each k and model like they did in the Codex paper [1].

[1] https://arxiv.org/abs/2107.03374

Post reply on HN