Earlier quoted context omitted.
It's 384 H100s for 24 days, costing less than half a million dollars.
H100 are going for about $3/hr, 384 24 3 ~ $28k
You can go much lower: https://gpulist.ai/
Smollm3: Smol, multilingual, long-context reasoner LLM
71–80 of 83 posts
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#72It's small (3B) and does great on benchmarks. This is a model for edge / mobile deployments so the gains over gemma3-4b are meaningful. It has dual mode reasoning / non_reasoning AND they released the full training method: > We're releasing SmolLM3 with our engineering blueprint. It includes architecture details, exact data mixtures showing how we progressively boost performance across domains in a three-stage pretra…
I hate to say it, but reasoning models simply aren't suited for edge computing. I just ran some tests on this model and even at 4bit weight quantisation it blows past 10GB of VRAM with just ~1000 tokens while it is still reasoning. So even if you're running on a dedicated ML edge device like a $250 Jetson, you will run out of memory before the model even formulates a real answer. You'll need a high end GPU to make full use of it for limited answers and an enterprise grade system to support longer contexts. And with reasoning turned off I don't see any meaningful improvement over older models.
So this is primarily great for enterprises who want to do on-prem with limited budgets and maybe high-end enthusiasts.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#73It's small (3B) and does great on benchmarks. This is a model for edge / mobile deployments so the gains over gemma3-4b are meaningful. It has dual mode reasoning / non_reasoning AND they released the full training method: > We're releasing SmolLM3 with our engineering blueprint. It includes architecture details, exact data mixtures showing how we progressively boost performance across domains in a three-stage pretra…
I hate to say it, but reasoning models simply aren't suited for edge computing. I just ran some tests on this model and even at 4bit weight quantisation it blows past 10GB of VRAM with just ~1000 tokens while it is still reasoning. So even if you're running on a dedicated ML edge device like a $250 Jetson, you will run out of memory before the model even formulates a real answer. You'll need a high end GPU to make fu…
You should use flash attention with KV cache quantization. I routinely use Qwen 3 14B with the full 128k context and it fits in under 24 GB VRAM. On my Pixel 8, I've successfully used Qwen 3 4B with 8K context (again with flash attention and KV cache quantization).
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#74Earlier quoted context omitted.
Hey Simon, VB from Hugging Face here and the person who added the model to MLX and llama.cpp (with Son). The PR hasn't yet landed on llama.cpp, hence it doesn't work OTB on llama.cpp installed via brew (similarly doesn't work with ollama since they need to bump their llama.cpp runtime) The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp If you want to avoid it, I added SmolLM3…
Could you please enlighten me regarding all these engines, I’m using lamacpp and ollama. Should I also try mlx, onnx, vllm, etc. I’m not quite sure whats the difference between all these. I’m running on CPU and sometimes GPU
Ollama is a wrapper around llama.cpp thei using ggml format. Onnx is different ml model format and onnxruntime developer by microsoft. Mlx is ml framework from Apple. If you want the fastest speed on MacOS most likely stick with mlx
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#75This is actually good learning material for anyone getting up to speed on LLM from scratch.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#76Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#77Earlier quoted context omitted.
I hate to say it, but reasoning models simply aren't suited for edge computing. I just ran some tests on this model and even at 4bit weight quantisation it blows past 10GB of VRAM with just ~1000 tokens while it is still reasoning. So even if you're running on a dedicated ML edge device like a $250 Jetson, you will run out of memory before the model even formulates a real answer. You'll need a high end GPU to make fu…
You should use flash attention with KV cache quantization. I routinely use Qwen 3 14B with the full 128k context and it fits in under 24 GB VRAM. On my Pixel 8, I've successfully used Qwen 3 4B with 8K context (again with flash attention and KV cache quantization).
>On my Pixel 8, I've successfully used Qwen 3 4B
How many tokens/s? I can't imagine that this would run in any practical way.
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#78I'm having trouble running this on my Mac - I've tried Ollama and llama.cpp llama-server so far, both using GGUFs from Hugging Face, but neither worked. (llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'smollm3') I've managed to run it using Python and transformers with PyTorch in device="cpu" mode but unsurprisingly that's really slow - it took 35s to respond to "…
Hey Simon, VB from Hugging Face here and the person who added the model to MLX and llama.cpp (with Son). The PR hasn't yet landed on llama.cpp, hence it doesn't work OTB on llama.cpp installed via brew (similarly doesn't work with ollama since they need to bump their llama.cpp runtime) The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp If you want to avoid it, I added SmolLM3…
> similarly doesn't work with ollama since they need to bump their llama.cpp runtime
Just curious, how frequently does that happen?
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#79Earlier quoted context omitted.
yes, both core web datasets are publicly available as well as the rest
Thanks! To be honest, if I might argue then that this is one of the best truly open source models that we have got. There is AllenAI and (Elmo?) and there is also this one which does distributed training but I think this looks a lot like SOTA for 3B parameters to me. Thanks for telling me, I am not going to lie, I am going to try to test it now! (Ima try some GGUF since ollama convenience)
there’s also IBM’s Granite
Re: Smollm3: Smol, multilingual, long-context reasoner LLM
#80I fixed some chat template issues for llama.cpp and other inference engines! To run it, do: ./llama.cpp/llama-cli -hf unsloth/SmolLM3-3B-GGUF:Q4_K_XL --jinja -ngl 99
> fixed some chat template issues This seems to be a persistent issue with almost all weight releases, even from bigger companies like Meta. Are the people who release these weights not testing them in various inference engines? Seems they make it work with Huggingface's Transformers library, then call it a day, but sometimes not even that.
Oh so chat template issues yes are quite pervasive sadly - for eg Llama as you mentioned, but also Qwen, Mistral, Google, the Phi team, DeepSeek - it's actually very common!
My take is large labs with closed source models also did have issues during the beginning, but most likely have standardized the chat template (for eg OpenAI using ChatML). The OSS community on the other hand keeps experimenting with new templates - for example adding tool calling causes a large headache. For example in https://unsloth.ai/blog/phi3 - we found many bugs in OSS models.