Tree Search Distillation for Language Models Using PPO
ayushtambde.com
Tree Search Distillation for Language Models Using PPO
1–10 of 15 posts
Re: Tree Search Distillation for Language Models Using PPO
#2[dead]
Re: Tree Search Distillation for Language Models Using PPO
#3[flagged]
Re: Tree Search Distillation for Language Models Using PPO
#4> One might note that MCTS uses more inference compute on a per-sample basis than GRPO: of course it performs better
This part confused me, it sounded like they were only doing the MCTS at train time, and then using GRPO to distill the MCTS policy into the model weights. So wouldn’t the model still have the same inference cost?
Re: Tree Search Distillation for Language Models Using PPO
#5Great post! I wonder why MCTS is not more popular as a test time compute harness. Did you compare performance of MCTS (without distillation) against other methods (eg best of N) with the same compute budget?
Re: Tree Search Distillation for Language Models Using PPO
#6> One might note that MCTS uses more inference compute on a per-sample basis than GRPO: of course it performs better This part confused me, it sounded like they were only doing the MCTS at train time, and then using GRPO to distill the MCTS policy into the model weights. So wouldn’t the model still have the same inference cost?
Ah, I meant that MCTS uses more inference-time compute (over GRPO) to produce a training sample
Re: Tree Search Distillation for Language Models Using PPO
#7[flagged]
Re: Tree Search Distillation for Language Models Using PPO
#8great write up (and effort!! ;))
what are your thoughts on MCTS for coding?
this can/must be paired with a smart execution harness to optimise roll out and roll back of execution paths and system state.
does this change the calculus for optimal post-training ?
Re: Tree Search Distillation for Language Models Using PPO
#9[deleted]
Re: Tree Search Distillation for Language Models Using PPO
#10Why is almost every RL paper done on Qwen-2.5 ? That decreases its credibility.