Smollm3: Smol, multilingual, long-context reasoner LLM
huggingface.co
Smollm3: Smol, multilingual, long-context reasoner LLM
1–10 of 83 posts
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#2It's small (3B) and does great on benchmarks. This is a model for edge / mobile deployments so the gains over gemma3-4b are meaningful. It has dual mode reasoning / non_reasoning AND they released the full training method:
> We're releasing SmolLM3 with our engineering blueprint. It includes architecture details, exact data mixtures showing how we progressively boost performance across domains in a three-stage pretraining approach, and the methodology for building a hybrid reasoning model. Usually, achieving these results would require months of reverse engineering. Instead, we're providing the full methodology.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#3Can anyone estimate how much of the 3B is necessitated by multi-language support?
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#4Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#5Which small model is good for fine tuning to various enterprise data sets? Our business units are wanting to run small models in browser and on mobile devices, without dealing with RAG and cloud resources.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#6Can anyone estimate how much of the 3B is necessitated by multi-language support?
The vocabulary size is fairly small (128,256) for a multilingual model. I would guess it doesn't require many additional parameters to support these 5 languages as many tokens can be shared.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#7Mostly SOTA performance at the 3B level. A notable addition to the small but truly open club of models that provide full disclosure, code, recipes to reproduce their work.
Looks like ballpark a million dollars of GPU time if you want to train up one for yourself (4000 gpus/24 days).
Very nice write up that’s generous in sharing their learnings.
This is a solid and positive contribution.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#8Which small model is good for fine tuning to various enterprise data sets? Our business units are wanting to run small models in browser and on mobile devices, without dealing with RAG and cloud resources.
You really need to try them all out yourself and make sure you have proper benchmarks.
While machine learning is not my field, I've tried to finetune Mistral 7B (following their official guide and toolset) and the results did not satisfy. Had a few very specific questions from the dataset that no matter how much I've finetuned and tweaked the process it was not able to respond with correct information.
A mix of vector search + keyword search is still better at building the right question context than expecting it to learn all the information.
I've used the pretrained dataset approach. Maybe building syntethic questions and answers around the dataset yields better results but I didn't have time to experiment with that approach.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#9There's a British comedy skit lurking in here.
"So it's a small large language model?"
"Oh yes, very small."
"How can it be small and large at the same time?"
"Well, it's small by the standards of a large language model."
"So it's large."
"Oh yes, very large."
"Large compared to what?"
"Small language models."
"And so something like ChatGPT, what would that be exactly? A large large language model?"
"Yes, precisely. An LLLM."
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#10Which small model is good for fine tuning to various enterprise data sets? Our business units are wanting to run small models in browser and on mobile devices, without dealing with RAG and cloud resources.
Small models are bad at knowing things. Trying to train knowledge in to small models is probably not the way you want to go. You could try building an offline embedded RAG system that is deployable as wasm. Some folks have been experiencing success with this.