What's up with this post? It's a link to something which has existed for a long time, and there's a bunch of dead comments below. Some weird SEO campaign thing?
Unsloth have just released benchmarks on how their dynamic quants perform for Qwen 3.5
Unsloth Dynamic 2.0 GGUFs
11–20 of 74 posts
Re: Unsloth Dynamic 2.0 GGUFs
#12What's up with this post? It's a link to something which has existed for a long time, and there's a bunch of dead comments below. Some weird SEO campaign thing?
Unsloth have just released benchmarks on how their dynamic quants perform for Qwen 3.5 https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks
I'm aware of that, but that's not the link of the post. The post is linking to their UD 2.0 quants from a few months back.
Also, the benchmarks are because they messed up the first version of their Qwen 3.5 XL quants by quanting some tensors to mxfp4 that should have been in higher quality, and this is their bugfix. The post literally starts out with "We updated Qwen3.5-35B Unsloth Dynamic quants being SOTA on nearly all bits" without explaining WHY they needed to update from the original version.
Re: Unsloth Dynamic 2.0 GGUFs
#13Cheers Daniel and Mike and team, keep up the good work!
Re: Unsloth Dynamic 2.0 GGUFs
#14ICYMI unsloth has had some major breakthroughs today with the Qwen3.5 local models https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks With the Qwen3.5 35B A3B at Q4 I've got 200k context running at 62.98 tokens per second on a local RTX5080 16GB.
Wait, the Q4 quantization which is more than 20GB fits in your 16GB GPU ? I didn't know that was possible, I was always restricting myself to smaller model than the VRAM I had
Yep. These Mixture of Experts models are well suited for paging in only the relevant data for a certain task https://huggingface.co/blog/moe
There's some experiments of just removing or merging experts post training to shrink models even more https://bknyaz.github.io/blog/2026/moe/
Re: Unsloth Dynamic 2.0 GGUFs
#15I run Llama 3.2 3B locally for latency-sensitive classification (sub-50ms, so no room for bigger models). At that scale Q2_K vs Q4_K_M isn't just smaller — Q2 starts flipping yes/no answers that Q4 gets right. Not often, but enough to notice in production.
So the KL divergence numbers here are more useful to me than the MMLU tables honestly. I've had MMLU hold steady while the output distribution drifted enough to break things downstream.
Does the calibration dataset make much difference at 3B though? There's so little redundancy that I'd expect it to hit a floor pretty fast regardless of how good the calibration data is.
Re: Unsloth Dynamic 2.0 GGUFs
#16What's up with this post? It's a link to something which has existed for a long time, and there's a bunch of dead comments below. Some weird SEO campaign thing?
Unsloth have just released benchmarks on how their dynamic quants perform for Qwen 3.5 https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks
Looking at their benchmarks there doesn't appear to be meaningful difference between their quants and bartowsky quants.
Re: Unsloth Dynamic 2.0 GGUFs
#17So q6 is practically perfect, and q3 is meaningfully decent. very impressive!
Re: Unsloth Dynamic 2.0 GGUFs
#18I run Llama 3.2 3B locally for latency-sensitive classification (sub-50ms, so no room for bigger models). At that scale Q2_K vs Q4_K_M isn't just smaller — Q2 starts flipping yes/no answers that Q4 gets right. Not often, but enough to notice in production. So the KL divergence numbers here are more useful to me than the MMLU tables honestly. I've had MMLU hold steady while the output distribution drifted enough to br…
For a simple classification task you generally want to prioritize regularization over more sophisticated behavior, so fewer parameters with larger quantization makes sense. For more generic chat-like purposes, Q2 of a larger model may often be preferable to Q4 of a smaller one.
Re: Unsloth Dynamic 2.0 GGUFs
#19I run Llama 3.2 3B locally for latency-sensitive classification (sub-50ms, so no room for bigger models). At that scale Q2_K vs Q4_K_M isn't just smaller — Q2 starts flipping yes/no answers that Q4 gets right. Not often, but enough to notice in production. So the KL divergence numbers here are more useful to me than the MMLU tables honestly. I've had MMLU hold steady while the output distribution drifted enough to br…
What do you use for sub-50ms inference?
Re: Unsloth Dynamic 2.0 GGUFs
#20This is pretty interesting, based on the blog post, it seems like they are using a technique similar to what I have been using to generate "layer sensitivity" data in my (still pretty beta) ggufy project, which is more aimed at diffusion (image) models.
https://github.com/qskousen/ggufy