[dead]
Open source inference time compute example from HuggingFace
21–28 of 28 posts
Re: Open source inference time compute example from HuggingFace
#22What's a point of such inference time compute if verifier is 8B model itself? Am I missing something?
I believe this is a valid point: HF's replication indeed uses larger off-the-shelf model as a verifier. In contrast, in the original paper, verifier is a fine-tune of the exact same base model which is used to sample step-by-step solutions (="solver").
Using different 1B model as verifier makes sense, yes. Using Llama 8B finetune as verifier to compare 1B inference time scaled in comparison with 8B makes little sense to me.
Using 3B model with 8B verifier against 70B model would make sense too. This being said their performance barely crossed 70B line with 256 examples. This is 256*(8+3)/70 ~ 40 times more computationally expensive than running 70B model as is.
Re: Open source inference time compute example from HuggingFace
#23What's a point of such inference time compute if verifier is 8B model itself? Am I missing something?
I believe this is a valid point: HF's replication indeed uses larger off-the-shelf model as a verifier. In contrast, in the original paper, verifier is a fine-tune of the exact same base model which is used to sample step-by-step solutions (="solver").
Where did you see that? I thought they used an 8b model for their reward model?
> To guide our search strategies, we used RLHFlow/Llama3.1-8B-PRM-Deepseek-Data, an 8B reward model that has been trained using process supervision
Re: Open source inference time compute example from HuggingFace
#24Earlier quoted context omitted.
I believe this is a valid point: HF's replication indeed uses larger off-the-shelf model as a verifier. In contrast, in the original paper, verifier is a fine-tune of the exact same base model which is used to sample step-by-step solutions (="solver").
Where did you see that? I thought they used an 8b model for their reward model? > To guide our search strategies, we used RLHFlow/Llama3.1-8B-PRM-Deepseek-Data, an 8B reward model that has been trained using process supervision
"Solver" is `meta-llama/Llama-3.2-1B-Instruct` (1B model, and they use 3B for another experiment), and verifier is `RLHFlow/Llama3.1-8B-PRM-Deepseek-Data`.
See https://github.com/huggingface/search-and-learn/blob/b3375f8... and https://github.com/huggingface/search-and-learn/blob/b3375f8...
In the original paper, they use PaLM 2-S* as "solver" and its fine-tune as "verifier".
Re: Open source inference time compute example from HuggingFace
#25Earlier quoted context omitted.
I believe this is a valid point: HF's replication indeed uses larger off-the-shelf model as a verifier. In contrast, in the original paper, verifier is a fine-tune of the exact same base model which is used to sample step-by-step solutions (="solver").
Using different 1B model as verifier makes sense, yes. Using Llama 8B finetune as verifier to compare 1B inference time scaled in comparison with 8B makes little sense to me. Using 3B model with 8B verifier against 70B model would make sense too. This being said their performance barely crossed 70B line with 256 examples. This is 256*(8+3)/70 ~ 40 times more computationally expensive than running 70B model as is.
"1B solver + 8B verifier + search" beating 0-shot 70B is nice, agree.
"1B solver + 8B verifier + search" beating 1B-0-shot or 1B-majority as baselines isn't illustrative imo. In other words, by using larger verifier, HF's replication fails to establish a "fair" baseline. Still an awesome blog and release/repository from HF's group - I love it!
Re: Open source inference time compute example from HuggingFace
#26The linked "bitter lesson" paper by Rich Sutton is so good!
Re: Open source inference time compute example from HuggingFace
#27Full blog is here: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling... Happy to answer any questions about these methods.
Great work! When I use models like o1, they work better than sonnet and 4o for tasks that require some thinking but the output is often very verbose. Is it possible to get the best of both worlds? The thinking takes place resulting in better performance but the output is straightforward to work with like with sonnet and 4o. Did you observe similar behaviour with the 1B and 3B models? How does the model behaviour chan…
OpenAI recommends using o1 to generate the verbose plan and then chain the verbose output to a cheaper model (e.g. gpt-4o-mini) to convert it into structured data / function calls / summary etc. They call it planner-executor pattern. [1]
Re: Open source inference time compute example from HuggingFace
#28Full blog is here: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling... Happy to answer any questions about these methods.
The big question is whether or not o3 is using any type of “meta-generation” algorithm at inference time, I.e are there multiple invocations of the LLM generation at all, or does it generate an insanely long reasoning trace in a single autoregressive stream that some somehow implicitly has search-like behavior? In other words, is the search-like behavior learned entirely in post-training and only implicitly exhibited at inference time, or is it explicitly done at inference time?
Given the enormous compute costs of o3, my speculation has been that search is explicit, but I’ve seen this post from Nathan Lambert for example that speculates (in the context of o1) that it’s possible for search to be entirely “baked-into” a single single stream roll-out (which would depend on significant long-context innovations):
https://www.interconnects.ai/p/openais-o1-using-search-was-a...
If true this would be extremely interesting.