Open source inference time compute example from HuggingFace
1–10 of 28 posts
Re: Open source inference time compute example from HuggingFace
#2Happy to see "inference time compute" term being used primarily nowadays - it's a much more precise and appropriate term compared to the unwieldy "test-time compute" that openai used to call it when they thought they "invented" scaling inference time.
Re: Open source inference time compute example from HuggingFace
#3Full blog is here: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling...
Happy to answer any questions about these methods.
Re: Open source inference time compute example from HuggingFace
#4Eli5?
Re: Open source inference time compute example from HuggingFace
#5Full blog is here: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling... Happy to answer any questions about these methods.
Excellent and interesting post!
Minor gripe - The best-of-n | beam search illustration is not compatible with red-green color blindness. I can literally not see the difference between the Rejected and the Selected dots even if I zoom in.
Re: Open source inference time compute example from HuggingFace
#6Full blog is here: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling... Happy to answer any questions about these methods.
Great work! When I use models like o1, they work better than sonnet and 4o for tasks that require some thinking but the output is often very verbose. Is it possible to get the best of both worlds? The thinking takes place resulting in better performance but the output is straightforward to work with like with sonnet and 4o. Did you observe similar behaviour with the 1B and 3B models? How does the model behaviour change when used for normal tasks that don't require thinking?
Also how well do these models work to extract structured output? Eg- perform ocr on some hand written text with math, convert to html and format formulas correctly etc. Single shot prompting doesn't work well with such problems but splitting the steps into consecutive api calls works well.
Re: Open source inference time compute example from HuggingFace
#7Full blog is here: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling... Happy to answer any questions about these methods.
Excellent and interesting post! Minor gripe - The best-of-n | beam search illustration is not compatible with red-green color blindness. I can literally not see the difference between the Rejected and the Selected dots even if I zoom in.
Thanks for the feedback, and not minor. Sorry about that.
Re: Open source inference time compute example from HuggingFace
#8Full blog is here: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling... Happy to answer any questions about these methods.
Great work! When I use models like o1, they work better than sonnet and 4o for tasks that require some thinking but the output is often very verbose. Is it possible to get the best of both worlds? The thinking takes place resulting in better performance but the output is straightforward to work with like with sonnet and 4o. Did you observe similar behaviour with the 1B and 3B models? How does the model behaviour chan…
That's a good point. We don't see that in our experiments because it's all in the math domain. However for OAI it's plausible that training for o1 might conflict with standard instruction training, leading to less human preferred output style.
Re: Open source inference time compute example from HuggingFace
#9Eli5?
For problems that require multi-step reasoning, standard LLMs seem to be stuck. The field is increasingly interested in models like o1 that output many "guesses" to find the right one. Currently open-source does not know how to do this, but we are reimplementing several possible directions to try. This replicates one important path using search and a verifier model.
Re: Open source inference time compute example from HuggingFace
#10Eli5?
If I understand correctly, Hugging Face is exploring approaches to tuning the output quality of a given model by tuning how long to let it run.
Normally when you run an LLM, you set your prompt and whatever tunable parameters, and the LLM software (eg. lamma.cpp) spits out tokens at whatever rate it can. If you want higher quality, you run a bigger model (though you're limited by the amount of memory you have available). If you want higher speed, you run a smaller model. Hugging Face seems to be looking at ways to make this tradeoff without switching between different models.