61–70 of 83 posts
Earlier quoted context omitted.
It's 384 H100s for 24 days, costing less than half a million dollars.
H100 are going for about $3/hr, 384 24 3 ~ $28k
Found this a few days ago which might be neat for finding cheaper https://www.primeintellect.ai/
No affiliation with either
I fixed some chat template issues for llama.cpp and other inference engines! To run it, do: ./llama.cpp/llama-cli -hf unsloth/SmolLM3-3B-GGUF:Q4_K_XL --jinja -ngl 99
> fixed some chat template issues This seems to be a persistent issue with almost all weight releases, even from bigger companies like Meta. Are the people who release these weights not testing them in various inference engines? Seems they make it work with Huggingface's Transformers library, then call it a day, but sometimes not even that.
The reason is simple. There isn't much money in it. llama.cpp is free and targets lower end of the hardware spectrum. Corporations will run something else, or even more likely, offload the task to contractor.
I'm having trouble running this on my Mac - I've tried Ollama and llama.cpp llama-server so far, both using GGUFs from Hugging Face, but neither worked. (llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'smollm3') I've managed to run it using Python and transformers with PyTorch in device="cpu" mode but unsurprisingly that's really slow - it took 35s to respond to "…
Hey Simon, VB from Hugging Face here and the person who added the model to MLX and llama.cpp (with Son). The PR hasn't yet landed on llama.cpp, hence it doesn't work OTB on llama.cpp installed via brew (similarly doesn't work with ollama since they need to bump their llama.cpp runtime) The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp If you want to avoid it, I added SmolLM3…
I've tried to use gemma3:4b which comes up better in that benchmark and found it to be quite disappointing. It breaks a lot, sucks even worse than qwen2.5-coder:7b and incept5/llama3.1-claude:7b at code, needs to be tricked or threatened into saying stuff about many everyday topics. It also commonly chugs away for minutes exercising the GPU fans before responding, at which point I'm already ahead because I figured ou…
Which small model is good for fine tuning to various enterprise data sets? Our business units are wanting to run small models in browser and on mobile devices, without dealing with RAG and cloud resources.
Bite the bullet and do some kind of RAG; you need to provide clear, authoritative information to a model that is skilled enough to remix it for the user.
Tuning the model to imitate the dataset will damage the model's skills and "common sense" but won't train it reliably recall information.
Earlier quoted context omitted.
It’s not locked behind a service though. https://huggingface.co/openai-community/gpt2/blob/main/model...
That’s only 124M param
Earlier quoted context omitted.
H100 are going for about $3/hr, 384 24 3 ~ $28k
This is indeed a reasonable cost estimate for competitive short-term H100 rentals (source: much SemiAnalysis coverage, and my own exploration of the market), but there is a critical error (besides the formatting glitch with `*`): It was 24 days (576 hours) not 24 hours. $663,552 @ $3/hr.
WARNING: This is highly speculative and napkin math
H200 (141 GB HBM3 - $3.99/h - 1.4x perf) 216 x 24 x 17 = 88128h = 351.895,104 (17 days and 216 cards)
B200 (192 GB HBM3e - $5.99/h - 2.8x perf) 158 x 24 x 9 = 34128h = $204.426,72
Probably wrong math, should be more efficient and cheaper. Doubt that they have 100/200 cards available for that long.
Source: I've only trained using RTX4090 and stuff like that with 8 cards.
Not affiliated in any way with Runpod.
Can anyone estimate how much of the 3B is necessitated by multi-language support?
Can anyone estimate how much of the 3B is necessitated by multi-language support?