21–30 of 482 posts
Earlier quoted context omitted.
My experience with qwen-3.6:35B-A3B reinforces this, gonna give this a spin when unsloth has quants available Gemini flash was just as good as pro for most tasks with good prompts, tools, and context. Gemma 4 was nearly as good as flash and Qwen 3.6 appears to be even better.
> when unsloth has quants available https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
This is why they don’t advertise which consumer hardware it can run on: Their direct release that delivers these results cannot fit on your average consumer system.
Most consumers don’t run the model they release directly. They run a quantized model that uses a lower number of bits per weight.
The quantizations come with tradeoffs. You will not get the exact results they advertise using a quantized version, but you can fit it on smaller hardware.
The previous 27B Qwen3.5 model had reasonable performance down to Q5 or Q4 depending on your threshold for quality loss. This was usable on a unified memory system (Mac, Strix Halo) with 32GB of extra RAM, so generally a 64GB Mac. They could also be run on an nVidia 5090 with 32GB RAM or a pair of 16GB or 24GB GPUs, which would not run as fast due to the split.
Watch out for some of the claims about running these models on iPhones or smaller systems. You can use a lot of tricks and heavy quantization to run it on very small systems but the quality of output will not be usable. There is a trend of posting “I ran this model and this small hardware” repos for social media bragging rights but the output isn’t actually good.
Unsloth quants available: https://unsloth.ai/docs/models/qwen3.6
Earlier quoted context omitted.
depends on format, compute type, quantization and kv cache size.
Specs for whatever they used to achieve the benchmarks would be a good start.
This will only run on server hardware, some workstation GPUs, or some 128GB unified memory systems.
It’s a situation where if you have to ask, you can’t run the exact model they released. You have to wait for quantizations to smaller sizes, which come in a lot of varieties and have quality tradeoffs.
I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
llama-server \
-hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \
--no-mmproj \
--fit on \
-np 1 \
-c 65536 \
--cache-ram 4096 -ctxcp 2 \
--jinja \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--reasoning on \
--chat-template-kwargs '{"preserve_thinking": true}'
I haven't tested the 27B model yet, but 35B-A3B often gets off rails after 15k-20k tokens of context. You can have it to do basic things reliably, but certainly not at the level of "frontier" models.
This is getting very close to fit a single 3090 with 24gb VRAM :)
Yup! Smaller quants will fit within 24GB but they might sacrifice context length. I’m excited to try out the MLX version to see if 32GB of memory from a Pro M-series Mac can get some acceptable tok/s with longer context. HuggingFace has uploaded some MLX versions already.
I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.