71–74 of 74 posts
What's up with this post? It's a link to something which has existed for a long time, and there's a bunch of dead comments below. Some weird SEO campaign thing?
Unsloth have just released benchmarks on how their dynamic quants perform for Qwen 3.5 https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks
I run Llama 3.2 3B locally for latency-sensitive classification (sub-50ms, so no room for bigger models). At that scale Q2_K vs Q4_K_M isn't just smaller — Q2 starts flipping yes/no answers that Q4 gets right. Not often, but enough to notice in production. So the KL divergence numbers here are more useful to me than the MMLU tables honestly. I've had MMLU hold steady while the output distribution drifted enough to br…
What do you use for sub-50ms inference?
Earlier quoted context omitted.
How much do you use? I have lots of trouble figuring out what the limits are of a system with x amount of vram and y amounts of ram. How do you determine this?
Ideally you'd have (parameter count) * (bits per parameter) VRAM for the entire (presumably quantized, don't forget to account for that) model. So very approximately 16 GiB for a 34B model quantized to 4 bits per parameter. You can spill to RAM in which case you at least want enough for a single active expert but really that's going to tank performance. If you're only "a bit" short of the full model the difference mi…
Earlier quoted context omitted.
I am aware of all that. They literally never say “they used mxfp4 in some weights”. What you’re claiming they said doesn’t exist. This isn’t a postmortem, it’s PR fluff without actually addressing the issue.
It's right there https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks I looked at the weights before. It's not PR fluff, they made it clear by showing how it really affected various tensors terribly. "MXFP4 is much worse on many tensors - attn_gate, attn_q, ssm_beta, ssm_alpha using MXFP4 is not a good idea, and rather Q4_K is better - also MXFP4 uses 4.25 bits per weight, whilst Q4_K uses 4.5 bits per weight. It's…
This is a famous non-apology non-explanation of what actually happened. "They made it clear by showing how it really affected various tensors terribly"? Where do they even say they had ever previously uploaded any quant with MXFP4?