I love the work unsloth is doing. I only wish gguf format had better vllm support. It’s sometimes hard to find trustworthy quants that work well with vllm.
Unsloth Dynamic 2.0 GGUFs
41–50 of 74 posts
Re: Unsloth Dynamic 2.0 GGUFs
#42ICYMI unsloth has had some major breakthroughs today with the Qwen3.5 local models https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks With the Qwen3.5 35B A3B at Q4 I've got 200k context running at 62.98 tokens per second on a local RTX5080 16GB.
Does llama.cpp support Qwen3.5 yet? When I tried it before, it failed saying "qwen35moe" is an unsupported architecture.
Yes, but make sure you grab the latest llama.cpp release
New model archs usually involve code changes.
Re: Unsloth Dynamic 2.0 GGUFs
#43Earlier quoted context omitted.
Does llama.cpp support Qwen3.5 yet? When I tried it before, it failed saying "qwen35moe" is an unsupported architecture.
Yes, but make sure you grab the latest llama.cpp release New model archs usually involve code changes.
Awesome! It looks like the llama.cpp-hip AUR was updated today to b8179, and it works.
Re: Unsloth Dynamic 2.0 GGUFs
#44Earlier quoted context omitted.
Wait, the Q4 quantization which is more than 20GB fits in your 16GB GPU ? I didn't know that was possible, I was always restricting myself to smaller model than the VRAM I had
Yep. These Mixture of Experts models are well suited for paging in only the relevant data for a certain task https://huggingface.co/blog/moe There's some experiments of just removing or merging experts post training to shrink models even more https://bknyaz.github.io/blog/2026/moe/
That blog post was super interesting. It is neat that he can select experts and control the routing in the model—not having played with the models in detail, tended to assume the “mixing” in mixture of experts was more like a blender, haha. The models are still quite lumpy I guess!
Re: Unsloth Dynamic 2.0 GGUFs
#45Great timing. I downloaded the models today on LM Studio, they seem to work remarkably well.
Any HN model recommendations to run on my 24GB M5 and any best practices while running them?
Re: Unsloth Dynamic 2.0 GGUFs
#46[dead]
Re: Unsloth Dynamic 2.0 GGUFs
#47ICYMI unsloth has had some major breakthroughs today with the Qwen3.5 local models https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks With the Qwen3.5 35B A3B at Q4 I've got 200k context running at 62.98 tokens per second on a local RTX5080 16GB.
2x RTX 4090, Q8, 256k context, 110 t/s
1 4090, Qwen3.5-35B-A3B-UD-MXFP4_MOE, 64k context, 122 t/s.
Llama.cpp
Re: Unsloth Dynamic 2.0 GGUFs
#48Earlier quoted context omitted.
Looking at their benchmarks there doesn't appear to be meaningful difference between their quants and bartowsky quants.
No our Qwen3.5 new ones show the opposite see https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks
Am I misreading the table?
Unsloth Q4_K_M
PPL: 6.6053 KLD 99.9%: 0.5478 KLD mean: 0.0192
bartowski Qwen_Q4_K_M
PPL: 6.6097 KLD 99.9%: 0.5771 KLD mean: 0.0182
Re: Unsloth Dynamic 2.0 GGUFs
#49Earlier quoted context omitted.
Wait, the Q4 quantization which is more than 20GB fits in your 16GB GPU ? I didn't know that was possible, I was always restricting myself to smaller model than the VRAM I had
Yep. These Mixture of Experts models are well suited for paging in only the relevant data for a certain task https://huggingface.co/blog/moe There's some experiments of just removing or merging experts post training to shrink models even more https://bknyaz.github.io/blog/2026/moe/
MoE is not suited for paging because it’s essentially a random expert per token. It only improves throughput because you reduce the memory bandwidth requirements for generating a token since 1/n of the weights are accessed per token (but a different 1/n on each loop).
Now shrinking them sure, but I’ve seen nothing that indicates you can just page weights in and out without cratering your performance like you would with a non MoE model
Re: Unsloth Dynamic 2.0 GGUFs
#50Earlier quoted context omitted.
Not really breakthroughs, more like bugfixes for their broken first batch.
No this is false - unsure if you saw our new blog - https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks which shows SOTA on nearly all bits, and we shared all our research as well
Explain what about that statement is false. Your original Q4_K_XL quant was broken. People noticing that it was a total outlier among other quants is what prompted this "research". Your own data proves that your new release fixes the bugs of your original, in order to match AesSedai's PPL. Fixing bugs is great. Searching for the best quant mix is helpful. I use your quants and appreciate your work. But whitewashing this situation dilutes trust and good will.