Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smollm3: Smol, multilingual, long-context reasoner LLM

huggingface.co

Post reply on HN
Print view View original

41–50 of 83 posts

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#41

Post by simonw »

I'm having trouble running this on my Mac - I've tried Ollama and llama.cpp llama-server so far, both using GGUFs from Hugging Face, but neither worked.

(llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'smollm3')

I've managed to run it using Python and transformers with PyTorch in device="cpu" mode but unsurprisingly that's really slow - it took 35s to respond to "say hi"!

Anyone had success with this on a Mac yet? I really want to get this running with tool calling, ideally via an OpenAI-compatible serving layer like llama-server.

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#42

Post by bitwize »

_kb wrote: post #38
bitwize wrote: post #9

There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…

Australian. This is straight up Clarke and Dawe / Utopia.

I must confess, I was inspired by "the front fell off".

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#44

Post by segmondy »

YetAnotherNick wrote: post #17
WhitneyLand wrote: post #7

Mostly SOTA performance at the 3B level. A notable addition to the small but truly open club of models that provide full disclosure, code, recipes to reproduce their work. Looks like ballpark a million dollars of GPU time if you want to train up one for yourself (4000 gpus/24 days). Very nice write up that’s generous in sharing their learnings. This is a solid and positive contribution.

It's 384 H100s for 24 days, costing less than half a million dollars.

H100 are going for about $3/hr, 384243 ~ $28k

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#46

Post by tripplyons »

simonw wrote: post #41

I'm having trouble running this on my Mac - I've tried Ollama and llama.cpp llama-server so far, both using GGUFs from Hugging Face, but neither worked. (llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'smollm3') I've managed to run it using Python and transformers with PyTorch in device="cpu" mode but unsurprisingly that's really slow - it took 35s to respond to "…

Have you tried setting device="mps" to use Metal? It should be faster than PyTorch's "cpu" device on Mac.

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#47

Post by viraptor »

_kb wrote: post #38
bitwize wrote: post #9

There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…

Australian. This is straight up Clarke and Dawe / Utopia.

"Yes, a British Australian comedy sketch."

"So it's British?"

"By heritage."

"But Australian?"

"By production."

"Ah, so it’s satire."

"It was, until someone funded it."

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#48

Post by jazzyjackson »

segmondy wrote: post #44
YetAnotherNick wrote: post #17

Earlier quoted context omitted.

It's 384 H100s for 24 days, costing less than half a million dollars.

H100 are going for about $3/hr, 384 24 3 ~ $28k

Take this brother, \*, it may serve you well

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#49

Post by danielhanchen »

segmondy wrote: post #45
danielhanchen wrote: post #37

I fixed some chat template issues for llama.cpp and other inference engines! To run it, do: ./llama.cpp/llama-cli -hf unsloth/SmolLM3-3B-GGUF:Q4_K_XL --jinja -ngl 99

doing the good work, thanks daniel!

Thank you!

Re: Smollm3: Smol, multilingual, long-context reasoner LLM

#50

Post by dr_kretyn »

segmondy wrote: post #44
YetAnotherNick wrote: post #17

Earlier quoted context omitted.

It's 384 H100s for 24 days, costing less than half a million dollars.

H100 are going for about $3/hr, 384 24 3 ~ $28k

The price just keeps on dropping with each comment. Anyone going to estimate it for less?

What's the source for $3/h?

Post reply on HN