Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 2.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

61–70 of 74 posts

Re: Unsloth Dynamic 2.0 GGUFs

#61

Post by danielhanchen »

lostmsu wrote: post #48
danielhanchen wrote: post #27

Earlier quoted context omitted.

No our Qwen3.5 new ones show the opposite see https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks

Am I misreading the table? Unsloth Q4_K_M PPL: 6.6053 KLD 99.9%: 0.5478 KLD mean: 0.0192 bartowski Qwen_Q4_K_M PPL: 6.6097 KLD 99.9%: 0.5771 KLD mean: 0.0182 Barely noticeable drop in PPL; noticeable KLD drop (good, 5%); but worse KLD mean (bad, 5%).

You forgot to check the disk sapce - _M and _XL are not the same across quants:

Unsloth Q4_K_M 18.49GB 0.5478 KLD 99.9% 0.0192 mean

Unsloth Q4_K_XL 19.17GB 0.4097 KLD 99.9% 0.0137 mean

bartowski Q4_K_M 19.77GB 0.5771 KLD 99.9% 0.0182 mean

Re: Unsloth Dynamic 2.0 GGUFs

#62

Post by p1esk »

cagenut wrote: post #58
vlovich123 wrote: post #57

Earlier quoted context omitted.

It’s called mixture of experts but it’s not that concepts map cleanly or even roughly to different experts. Otherwise you wouldn’t get a new expert on every token. You have to remember these were designed to improve throughput in cloud deployments where different GPUs load an expert. There you precisely want each expert to handle randomly to improve your GPU utilization rate. I have not heard anyone training local Mo…

is there anywhere good to read/follow to get operational clarity on this stuff? my current system of looking for 1 in 1000 posts on HN or 1 in 100 on r/locallama is tedious.

Ask any of the models to explain this to you

Re: Unsloth Dynamic 2.0 GGUFs

#63

Post by mirekrusin »

instagib wrote: post #47
mirekrusin wrote: post #21

Earlier quoted context omitted.

2x RTX 4090, Q8, 256k context, 110 t/s

1 4090, Qwen3.5-35B-A3B-UD-MXFP4_MOE, 64k context, 122 t/s. Llama.cpp

I believe it's mentioned that MXFP4 performs surprisingly bad, you may want to try other Q4s.

Re: Unsloth Dynamic 2.0 GGUFs

#64

Post by pyuser583 »

segmondy wrote: post #7
Kayou wrote: post #6

Earlier quoted context omitted.

Wait, the Q4 quantization which is more than 20GB fits in your 16GB GPU ? I didn't know that was possible, I was always restricting myself to smaller model than the VRAM I had

llama.cpp is designed for partial offloading, the most important part of the model will be loaded into the GPU and the rest on system ram. I run 500B+ models such as DeepSeek/KimiK2.5/GLM-5 without having that much GPU vram.

How much do you use?

I have lots of trouble figuring out what the limits are of a system with x amount of vram and y amounts of ram. How do you determine this?

Re: Unsloth Dynamic 2.0 GGUFs

#65

Post by oofbey »

What does it mean to say that “99.9% KL divergence” is some number like 3? In AI research and math, KL divergence is a pseudo-distance metric from one distribution to another. (Not technically a distance between two distributions because it’s asymmetric.)

Folks here who spend lots of time thinking about compressing models apparently have some specific interpretation of the term. Can somebody educate me? Because I only Understand the math definition.

Re: Unsloth Dynamic 2.0 GGUFs

#66

Post by segmondy »

jychang wrote: post #56
segmondy wrote: post #55

Earlier quoted context omitted.

each layer is made up of various weights, the weights are adjusted to quant it. a pure q8 will have all the weights as q8, or a q4 the same. but some are kept as f32, etc. here's an example of q3_k_xl - https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF/tree/ma... we can see certain weights are f32, q8, q5, q3, etc. They used mxfp4 in some weights and mxfp4 doesn't seem to place nicely in quants so that's why they…

I am aware of all that. They literally never say “they used mxfp4 in some weights”. What you’re claiming they said doesn’t exist. This isn’t a postmortem, it’s PR fluff without actually addressing the issue.

It's right there https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks I looked at the weights before. It's not PR fluff, they made it clear by showing how it really affected various tensors terribly.

"MXFP4 is much worse on many tensors - attn_gate, attn_q, ssm_beta, ssm_alpha using MXFP4 is not a good idea, and rather Q4_K is better - also MXFP4 uses 4.25 bits per weight, whilst Q4_K uses 4.5 bits per weight. It's better to use Q4_K than MXFP4 when choosing between them."

The Q4 quants had a mixture of mxfp4 leading to worse outcomes.

Re: Unsloth Dynamic 2.0 GGUFs

#68

Post by fc417fc802 »

pyuser583 wrote: post #64
segmondy wrote: post #7

Earlier quoted context omitted.

llama.cpp is designed for partial offloading, the most important part of the model will be loaded into the GPU and the rest on system ram. I run 500B+ models such as DeepSeek/KimiK2.5/GLM-5 without having that much GPU vram.

How much do you use? I have lots of trouble figuring out what the limits are of a system with x amount of vram and y amounts of ram. How do you determine this?

Ideally you'd have (parameter count) * (bits per parameter) VRAM for the entire (presumably quantized, don't forget to account for that) model. So very approximately 16 GiB for a 34B model quantized to 4 bits per parameter.

You can spill to RAM in which case you at least want enough for a single active expert but really that's going to tank performance. If you're only "a bit" short of the full model the difference might not be all that large.

These things are memory bandwidth limited so if you check out RAM, VRAM, and PCIe bandwidth what I wrote above should make sense.

Also you should just ask your friendly local LLM these sorts of questions.

Re: Unsloth Dynamic 2.0 GGUFs

#69

Post by lostmsu »

danielhanchen wrote: post #61
lostmsu wrote: post #48

Earlier quoted context omitted.

Am I misreading the table? Unsloth Q4_K_M PPL: 6.6053 KLD 99.9%: 0.5478 KLD mean: 0.0192 bartowski Qwen_Q4_K_M PPL: 6.6097 KLD 99.9%: 0.5771 KLD mean: 0.0182 Barely noticeable drop in PPL; noticeable KLD drop (good, 5%); but worse KLD mean (bad, 5%).

You forgot to check the disk sapce - _M and _XL are not the same across quants: Unsloth Q4_K_M 18.49GB 0.5478 KLD 99.9% 0.0192 mean Unsloth Q4_K_XL 19.17GB 0.4097 KLD 99.9% 0.0137 mean bartowski Q4_K_M 19.77GB 0.5771 KLD 99.9% 0.0182 mean

The table doesn't have bartowski Q4_K_XL to compare, but given the metrics of _Ms aren't universally better it's unclear if smaller size doesn't come with a cost.

Re: Unsloth Dynamic 2.0 GGUFs

#70

Post by oofbey »

oofbey wrote: post #65

What does it mean to say that “99.9% KL divergence” is some number like 3? In AI research and math, KL divergence is a pseudo-distance metric from one distribution to another. (Not technically a distance between two distributions because it’s asymmetric.) Folks here who spend lots of time thinking about compressing models apparently have some specific interpretation of the term. Can somebody educate me? Because I onl…

The confusing thing here is that there are two distributions involved here. There's the distribution over the vocabulary (possible values of each token) and the distribution over the sequence of tokens in each document.

Here, the KL Divergence is calculated over the vocabulary's distribution - for a specific token, it is measuring how much the quantized model's predictions differ from the reference model. 0 means a perfect match (no loss of quality from quantizaton), and some large number like 4 nats meaning the quantized model's predictions for that token differ substantially from the reference model.

The 99.9% is taken over the sequence of tokens. So it ranks all the tokens in a corpus, and it effectively finds the token with the worst predictions (relative to the reference model) out of every 1000 tokens. That's the 99.9%ile part.

Post reply on HN