Long-Context GRPO
unsloth.ai
Long-Context GRPO
1–10 of 23 posts
Re: Long-Context GRPO
#2Oh thanks for posting! If anyone has any questions about stuff, feel free to ask!
Re: Long-Context GRPO
#3This is what I understood from the blog post (please correct me if I am wrong):
Unsloth allows you to give it a transformer model and additional training data to do LoRA/QLoRA. LoRA/QLoRA will keep the weights of the model as constant but as output some low rank adjustments to the weights which serves as the weight "delta".
Typically one would do SFT with the training data. But Unsloth allows you to do RL (Reinforcement learning) specifically GRPO on the model + training data you give it also! The output of the GRPO here is again in the form the LoRA/QLoRA weights.
You have found a way to reduce the memory requirements for GRPO.
Question: How does one decide whether the training data will be SFT (Supervised fine tuning) or GRPO ? When will you get better results with SFT and when with GRPO ?
Re: Long-Context GRPO
#4This is what I understood from the blog post (please correct me if I am wrong): Unsloth allows you to give it a transformer model and additional training data to do LoRA/QLoRA. LoRA/QLoRA will keep the weights of the model as constant but as output some low rank adjustments to the weights which serves as the weight "delta". Typically one would do SFT with the training data. But Unsloth allows you to do RL (Reinforcem…
Yes you're correct!
Very good question on SFT vs GRPO!
Assume the dataset I have is "What is 2+2?", "The answer is 4".
1. If you have very high quality labelled data, SFT should work fine. Ie "What is 2+2? Let me think about it....., The Answer is 4"
2. If you only have the input "What is 2+2", and just the answer "4", but nothing in between, GRPO could be very helpful! GRPO can help produce the reasoning traces automatically - you will need to provide some scoring / reward functions though. For example if the answer == 4, + 1 score.
3. You can combine SFT and GRPO! Do SFT first, then GRPO - this actually makes GRPO most likely converge faster!
Re: Long-Context GRPO
#5This is what I understood from the blog post (please correct me if I am wrong): Unsloth allows you to give it a transformer model and additional training data to do LoRA/QLoRA. LoRA/QLoRA will keep the weights of the model as constant but as output some low rank adjustments to the weights which serves as the weight "delta". Typically one would do SFT with the training data. But Unsloth allows you to do RL (Reinforcem…
Yes you're correct! Very good question on SFT vs GRPO! Assume the dataset I have is "What is 2+2?", "The answer is 4". 1. If you have very high quality labelled data, SFT should work fine. Ie "What is 2+2? Let me think about it....., The Answer is 4" 2. If you only have the input "What is 2+2", and just the answer "4", but nothing in between, GRPO could be very helpful! GRPO can help produce the reasoning traces auto…
Does this mean that you can only do GRPO on the training models that have reasoning traces in ...
Re: Long-Context GRPO
#6Earlier quoted context omitted.
Yes you're correct! Very good question on SFT vs GRPO! Assume the dataset I have is "What is 2+2?", "The answer is 4". 1. If you have very high quality labelled data, SFT should work fine. Ie "What is 2+2? Let me think about it....., The Answer is 4" 2. If you only have the input "What is 2+2", and just the answer "4", but nothing in between, GRPO could be very helpful! GRPO can help produce the reasoning traces auto…
Does this mean that you can only do GRPO on the training models that have reasoning traces in ...
Oh no at all!! You can actually convert a model to even generate the ... tokens themselves! That's how DeepSeek trained R1 Zero, which essentially made the model have reasoning skills!
Re: Long-Context GRPO
#7This is what I understood from the blog post (please correct me if I am wrong): Unsloth allows you to give it a transformer model and additional training data to do LoRA/QLoRA. LoRA/QLoRA will keep the weights of the model as constant but as output some low rank adjustments to the weights which serves as the weight "delta". Typically one would do SFT with the training data. But Unsloth allows you to do RL (Reinforcem…
Yes you're correct! Very good question on SFT vs GRPO! Assume the dataset I have is "What is 2+2?", "The answer is 4". 1. If you have very high quality labelled data, SFT should work fine. Ie "What is 2+2? Let me think about it....., The Answer is 4" 2. If you only have the input "What is 2+2", and just the answer "4", but nothing in between, GRPO could be very helpful! GRPO can help produce the reasoning traces auto…
can you give some real-world examples for when this would be useful? Does this work for tasks requiring tool calling as well?
Re: Long-Context GRPO
#8Earlier quoted context omitted.
Yes you're correct! Very good question on SFT vs GRPO! Assume the dataset I have is "What is 2+2?", "The answer is 4". 1. If you have very high quality labelled data, SFT should work fine. Ie "What is 2+2? Let me think about it....., The Answer is 4" 2. If you only have the input "What is 2+2", and just the answer "4", but nothing in between, GRPO could be very helpful! GRPO can help produce the reasoning traces auto…
can you give some real-world examples for when this would be useful? Does this work for tasks requiring tool calling as well?
Yes tool calling is a prime example!! Ie you have some specific task, and the final output involving some tools, but sadly the steps to call the tools / the stuff in between / the thinking process is missing.
You can employ GRPO and maybe add an actual Python environment for the model to learn to act in.
Re: Long-Context GRPO
#9Earlier quoted context omitted.
Does this mean that you can only do GRPO on the training models that have reasoning traces in ...
Oh no at all!! You can actually convert a model to even generate the ... tokens themselves! That's how DeepSeek trained R1 Zero, which essentially made the model have reasoning skills!
Wont you have to use a distilled DeepThink model then ? Because the training phase with GRPO required to its reasoning within for least loss.
Re: Long-Context GRPO
#10Earlier quoted context omitted.
Oh no at all!! You can actually convert a model to even generate the ... tokens themselves! That's how DeepSeek trained R1 Zero, which essentially made the model have reasoning skills!
Wont you have to use a distilled DeepThink model then ? Because the training phase with GRPO required to its reasoning within for least loss.
Oh no no!! The trick for GRPO is you essentially let the model "learn" how to do reasoning itself!!!
The tokens are optional for formatting reasons. You could use or or [reasoning] for example in the system prompt.