Long-Context GRPO • Hacker News Board

Re: Long-Context GRPO

#21

Post by danielhanchen » Sat, Feb 22, 2025, 12:03 AM UTC

codelion wrote: post #13

sidkshatriya wrote: post #5

Earlier quoted context omitted.

Does this mean that you can only do GRPO on the training models that have reasoning traces in ...

Models already have hidden latent CoT style reasoning within them, GRPO would help induce that behavior. For instance see https://x.com/asankhaya/status/1838375748165628053 where a sampling technique (CoT decoding) can actual improve performance of the model.

Oh yep! The deepseek paper also mentioned how large enough LLMs inherently have responding capabilities and the goal of GRPO is to accentuate latent skills!

Re: Long-Context GRPO

#22

Post by danielhanchen » Sat, Feb 22, 2025, 12:04 AM UTC

imjonse wrote: post #18

sidkshatriya wrote: post #3

This is what I understood from the blog post (please correct me if I am wrong): Unsloth allows you to give it a transformer model and additional training data to do LoRA/QLoRA. LoRA/QLoRA will keep the weights of the model as constant but as output some low rank adjustments to the weights which serves as the weight "delta". Typically one would do SFT with the training data. But Unsloth allows you to do RL (Reinforcem…

Is it established whether GRPO is essential for this to work as it does, or could other RLHF-class methods provide similar results? My initial (possibly mistaken) impression was that GRPO was one of ways of mitigating the lack of enormous hardware resources.

Yep so GRPO is much more memory efficient than PPO, but other RL type algorithms can work fine as well!

Re: Long-Context GRPO

#23

Post by danielhanchen » Sat, Feb 22, 2025, 7:27 AM UTC

loxias wrote: post #17

danielhanchen wrote: post #2

Oh thanks for posting! If anyone has any questions about stuff, feel free to ask!

Thanks for what you're doing. Of all the various companies and orgs posting chatter about deep learning, I've come to really appreciate your efforts (and Anthropic), because you're USING MATH. :) I have some understanding of applied math, continuous and discrete, and while I don't keep up to date with developments in deep learning/AI in general, I always look forward to unsloth posts because they tend to center on ac…

Oh thank you a lot!!

I always keep maths at the center of everything we do :) It's literally humanity's ultimate superpower if we can express everything in mathematical terms!

I'll keep writing up blog posts with more maths!!