Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

arxiv.org

Post reply on HN
Print view View original

21–30 of 34 posts

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#21

Post by t55 »

yorwba wrote: post #16
t55 wrote: post #8

Earlier quoted context omitted.

> prolonged RL training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling does this mean that previous RL papers claiming the opposite were possibly bottlenecked by small datasets?

No, they do not point to any specific examples of novel reasoning strategies that were uncovered, nor is their sampling that extensive (at most 256 samples vs the 2048 used in https://limit-of-rlvr.github.io/ ).

so you think it's fake news? another example of a paper with strong claims without much evidence?

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#22

Post by t55 »

jimmySixDOF wrote: post #10

RL is proving to be a weird science lately : >Spurious Rewards: Rethinking Training Signals in RLVR ### *TL;DR* We show that you can do RLVR on Qwen2.5-Math models with *completely random or incorrect rewards*, and still get massive math benchmark gains. All of the following spurious rewards give 15-20+ points on MATH-500 when RLVR training Qwen2.5-Math-7B: - RLVR + format reward (reward responses with `\boxed{}`): *…

yeah, RLVR is still nascent and hence there's lots of noise.

> How can these spurious rewards possibly work? Can we get similar gains on other models with broken rewards?

it's because in those cases, RLVR merely elicits the reasoning strategies already contained in the model through pre-training

this paper, which uses Reasoning gym, shows that you need to train for way longer than those papers you mentioned to actually uncover novel reasoning strategies: https://arxiv.org/abs/2505.24864

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#23

Post by t55 »

flowerthoughts wrote: post #9
t55 wrote: post #5

Earlier quoted context omitted.

> I personally think that Gemini 2.5 Pro's superiority comes from having hundreds or thousands RL tasks (without any proof whatsoever, so rather a feeling). Given that GDM pioneered RL, that's a reasonable assumption

Assuming with GDM, you mean Google-Deep Mind. They pioneered RL with deep nets as policy function estimator. The deep nets being a result of CNNs and massive improvements in hardware parallelization at the time. RL was established, at the latest, with Q-learning in 1989: https://en.wikipedia.org/wiki/Q-learning

i didn't say they invented everything; in science you always stand on the shoulders of giants

i still think my original statement is fair

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#24

Post by spmurrayzzz »

jimmySixDOF wrote: post #10

RL is proving to be a weird science lately : >Spurious Rewards: Rethinking Training Signals in RLVR ### *TL;DR* We show that you can do RLVR on Qwen2.5-Math models with *completely random or incorrect rewards*, and still get massive math benchmark gains. All of the following spurious rewards give 15-20+ points on MATH-500 when RLVR training Qwen2.5-Math-7B: - RLVR + format reward (reward responses with `\boxed{}`): *…

I think the fact that spurious rewards were predominantly only effective for Qwen may suggest that it was triggering some shift in its language distribution. If you use those models long enough you'll see a ton of mandarin that makes its way into your outputs, and their logits tend to look more "confident" than the ones for english tokens.

So the reward value shifting may act as a sort of unintentional regularization technique (similar to adding noise to the discriminator input in GAN archs).

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#25

Post by t55 »

CuriouslyC wrote: post #12
phh wrote: post #4

Cool cool. I'm a bit put off by calling it "reasoning" /"thought". These RL targets can be achieved without "thinking" model but still cool. Gotta love the brainfuck task. I personally think that Gemini 2.5 Pro's superiority comes from having hundreds or thousands RL tasks (without any proof whatsoever, so rather a feeling). So I've been wanting a "RL Zoo" for quite a while. I hope this project won't be a one-off and…

Gemini 2.5 Pro's superiority is IMO largely driven by their long context support and training methodology. Compare Gemini as a beta reader for a 100k token book with GPT4.1 or Claude 4, and it becomes quite clear how much more effectively it can reason across its context than other comparable models. This also makes it much better for architecting new features into a system, since you can load a lot of the current sy…

For a 100k token context window; all those models are comparable though

gemini 2.5 pro shines for 200k+ tokens

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#27

Post by CuriouslyC »

t55 wrote: post #25
CuriouslyC wrote: post #12

Earlier quoted context omitted.

Gemini 2.5 Pro's superiority is IMO largely driven by their long context support and training methodology. Compare Gemini as a beta reader for a 100k token book with GPT4.1 or Claude 4, and it becomes quite clear how much more effectively it can reason across its context than other comparable models. This also makes it much better for architecting new features into a system, since you can load a lot of the current sy…

For a 100k token context window; all those models are comparable though gemini 2.5 pro shines for 200k+ tokens

I can confirm from first hand experience that even at 100k they are most definitely not comparable for the task of beta reading.

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#28

Post by throwaway314155 »

t55 wrote: post #25
CuriouslyC wrote: post #12

Earlier quoted context omitted.

Gemini 2.5 Pro's superiority is IMO largely driven by their long context support and training methodology. Compare Gemini as a beta reader for a 100k token book with GPT4.1 or Claude 4, and it becomes quite clear how much more effectively it can reason across its context than other comparable models. This also makes it much better for architecting new features into a system, since you can load a lot of the current sy…

For a 100k token context window; all those models are comparable though gemini 2.5 pro shines for 200k+ tokens

splitting hairs much?

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#29

Post by yorwba »

t55 wrote: post #21
yorwba wrote: post #16

Earlier quoted context omitted.

No, they do not point to any specific examples of novel reasoning strategies that were uncovered, nor is their sampling that extensive (at most 256 samples vs the 2048 used in https://limit-of-rlvr.github.io/ ).

so you think it's fake news? another example of a paper with strong claims without much evidence?

I think it's a case of not coming up with alternative explanations for the observed evidence and hence not designing experiments to distinguish between those explanations.

Their results are consistent with novel reasoning strategies, but they're also consistent with more reliable execution of reasoning strategies that the base model can generate in principle, but rarely succeeds at due to a large number of steps. (If you have a model that can do each step independently with 99% success rate and getting the correct result requires 1000 steps, the chance of making it all the way to the end without a single error is only about 0.004%.)

Re: ReasoningGym: Reasoning Environments for RL with Verifiable Rewards

#30

Post by psb217 »

yorwba wrote: post #29
t55 wrote: post #21

Earlier quoted context omitted.

so you think it's fake news? another example of a paper with strong claims without much evidence?

I think it's a case of not coming up with alternative explanations for the observed evidence and hence not designing experiments to distinguish between those explanations. Their results are consistent with novel reasoning strategies, but they're also consistent with more reliable execution of reasoning strategies that the base model can generate in principle, but rarely succeeds at due to a large number of steps. (If…

One challenge with this line of argument is that the base model assigns non-zero probability to all possible sequences if we ignore truncation due to numerical precision. So, in a sense you could say any performance improvement is due to shifting probability mass towards good reasoning behaviors and away from bad ones that were already present in the base model.

I agree with your general point though. Ie, we need more thorough empirical investigation of how reasoning behavior evolves during RL training starting from the base model. And, current RL training results seem more like "amplifying existing good behavior" than "inducing emergent good behavior".

Post reply on HN