51–60 of 83 posts
Earlier quoted context omitted.
It's 384 H100s for 24 days, costing less than half a million dollars.
H100 are going for about $3/hr, 384 24 3 ~ $28k
It was 24 days (576 hours) not 24 hours. $663,552 @ $3/hr.
Earlier quoted context omitted.
H100 are going for about $3/hr, 384 24 3 ~ $28k
The price just keeps on dropping with each comment. Anyone going to estimate it for less? What's the source for $3/h?
There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…
My experience with phi4-mini and granite3.3 was about the same, and they annoy me even more when I hook them into code editors and try to get them to contribute to my work. For one because they're slow, and at best they suggest adding unnecessary error handling in the style of null checks everywhere, at worst they just start mixing or hallucinating programming languages. Where they would be useful as leverage if they worked, i.e. close to the edge of where I can debug and refactor without getting stuck, they just go into straight nonsense mode, especially on terse first-pass code.
Sometimes I've tried to query these things for descriptions of recent history in foreign countries, Wikipedia trivia basically, and they're very often wrong in subtle ways. For example, a politician might have been at it for half a century or so in a troubled country and because they've been ousted in a coup once in the eighties the model is absolutely sure they can't have been in office since.
If a person acted like these things do I'd wish for them to get immediate institutional care. Maybe the problem is somehow with me, but I have a deep suspicion it's not.
There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…
Standards have shifted as well. Gpt2 used to be considered “large” but it is half the size of this. Oh and also Sam Altman said it was too dangerous to release. At this point I consider anything too big to run on consumer grade hardware to be large, but an exact definition is a little silly to argue about.
Earlier quoted context omitted.
It's 384 H100s for 24 days, costing less than half a million dollars.
H100 are going for about $3/hr, 384 24 3 ~ $28k
Earlier quoted context omitted.
Thanks! To be honest, if I might argue then that this is one of the best truly open source models that we have got. There is AllenAI and (Elmo?) and there is also this one which does distributed training but I think this looks a lot like SOTA for 3B parameters to me. Thanks for telling me, I am not going to lie, I am going to try to test it now! (Ima try some GGUF since ollama convenience)
OLMo: https://allenai.org/olmo AFAIK, they were the first open everything model.
GPT2 (released ~5 years ago?) was "open" in the sense that weights were available for download (sans license), exact datasets that were used where outlined, the architecture explained and so on, so I guess it was also "open" in the sense that Llama is "open", but neither would be "open source" which I'd feel pretty confident to label OLMo with.
So OLMo seems to be the first actually "open source" model, but maybe not "open" as in "downloadable" (which Facebook tries to call "open source").
I fixed some chat template issues for llama.cpp and other inference engines! To run it, do: ./llama.cpp/llama-cli -hf unsloth/SmolLM3-3B-GGUF:Q4_K_XL --jinja -ngl 99
This seems to be a persistent issue with almost all weight releases, even from bigger companies like Meta.
Are the people who release these weights not testing them in various inference engines? Seems they make it work with Huggingface's Transformers library, then call it a day, but sometimes not even that.
I'm having trouble running this on my Mac - I've tried Ollama and llama.cpp llama-server so far, both using GGUFs from Hugging Face, but neither worked. (llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'smollm3') I've managed to run it using Python and transformers with PyTorch in device="cpu" mode but unsurprisingly that's really slow - it took 35s to respond to "…
The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp
If you want to avoid it, I added SmolLM3 to MLX-LM as well:
You can run it via `mlx_lm.chat --model "mlx-community/SmolLM3-3B-bf16"`
(requires the latest mlx-lm to be installed)
here's the MLX-lm PR if you're interested: https://github.com/ml-explore/mlx-lm/pull/272
similarly, llama.cpp here: https://github.com/ggml-org/llama.cpp/pull/14581
Let me know if you face any issues!