Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 2.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

51–60 of 74 posts

Re: Unsloth Dynamic 2.0 GGUFs

#51

Post by hedora »

danielhanchen wrote: post #24
Maxious wrote: post #3

ICYMI unsloth has had some major breakthroughs today with the Qwen3.5 local models https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks With the Qwen3.5 35B A3B at Q4 I've got 200k context running at 62.98 tokens per second on a local RTX5080 16GB.

Oh I didn't expect this to be on HN haha - but yes for our new benchmarks for Qwen3.5, we devised a slightly different approach for quantization which we plan to roll out to all new models from now on!

Nice! Your stuff ran LLMs extremely well on I’m eager to try it out, especially if 16GB is viable now.

Re: Unsloth Dynamic 2.0 GGUFs

#52

Post by FuckButtons »

vlovich123 wrote: post #49
Maxious wrote: post #14

Earlier quoted context omitted.

Yep. These Mixture of Experts models are well suited for paging in only the relevant data for a certain task https://huggingface.co/blog/moe There's some experiments of just removing or merging experts post training to shrink models even more https://bknyaz.github.io/blog/2026/moe/

MoE is not suited for paging because it’s essentially a random expert per token. It only improves throughput because you reduce the memory bandwidth requirements for generating a token since 1/n of the weights are accessed per token (but a different 1/n on each loop). Now shrinking them sure, but I’ve seen nothing that indicates you can just page weights in and out without cratering your performance like you would wi…

Not entirely true, it’s random access within the relevant subset of experts and since concepts are clustered you actually have a much higher probability of repeatedly accessing the same subset of experts more frequently.

Re: Unsloth Dynamic 2.0 GGUFs

#55

Post by segmondy »

jychang wrote: post #35
danielhanchen wrote: post #28

Earlier quoted context omitted.

Didn't expect this to be on HN haha - but sometimes HN does have older posts come up sometimes. No your conclusion is false - only the old Q4_K_XL had slightly higher perplexity, all other quants are fine. We uploaded 9TB of research artifacts to https://huggingface.co/unsloth/Qwen3.5-35B-A3B-Experiments-G... for the community. If you read our blog, it says KLD and PPL are actually sometimes counterintuitive - for ex…

The Q4_K_XL is easily the most popular quant for the model, though. So then why was Q4_K_XL having issues? Is it just a PPL issue that doesn't reflect in real world usage? If yes, why not just say that? "The Q4_K_XL had lower PPL, but don't worry, PPL can be wrong, and other benchmarks show it's fine". If it was a real quality issue, then where was the issue caused by? The blog post says "Retiring MXFP4 from all GGUF…

each layer is made up of various weights, the weights are adjusted to quant it. a pure q8 will have all the weights as q8, or a q4 the same. but some are kept as f32, etc. here's an example of q3_k_xl - https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF/tree/ma... we can see certain weights are f32, q8, q5, q3, etc. They used mxfp4 in some weights and mxfp4 doesn't seem to place nicely in quants so that's why they are retiring it. read their publication again and it should make more sense.

Re: Unsloth Dynamic 2.0 GGUFs

#56

Post by jychang »

segmondy wrote: post #55
jychang wrote: post #35

Earlier quoted context omitted.

The Q4_K_XL is easily the most popular quant for the model, though. So then why was Q4_K_XL having issues? Is it just a PPL issue that doesn't reflect in real world usage? If yes, why not just say that? "The Q4_K_XL had lower PPL, but don't worry, PPL can be wrong, and other benchmarks show it's fine". If it was a real quality issue, then where was the issue caused by? The blog post says "Retiring MXFP4 from all GGUF…

each layer is made up of various weights, the weights are adjusted to quant it. a pure q8 will have all the weights as q8, or a q4 the same. but some are kept as f32, etc. here's an example of q3_k_xl - https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF/tree/ma... we can see certain weights are f32, q8, q5, q3, etc. They used mxfp4 in some weights and mxfp4 doesn't seem to place nicely in quants so that's why they…

I am aware of all that.

They literally never say “they used mxfp4 in some weights”. What you’re claiming they said doesn’t exist.

This isn’t a postmortem, it’s PR fluff without actually addressing the issue.

Re: Unsloth Dynamic 2.0 GGUFs

#57

Post by vlovich123 »

FuckButtons wrote: post #52
vlovich123 wrote: post #49

Earlier quoted context omitted.

MoE is not suited for paging because it’s essentially a random expert per token. It only improves throughput because you reduce the memory bandwidth requirements for generating a token since 1/n of the weights are accessed per token (but a different 1/n on each loop). Now shrinking them sure, but I’ve seen nothing that indicates you can just page weights in and out without cratering your performance like you would wi…

Not entirely true, it’s random access within the relevant subset of experts and since concepts are clustered you actually have a much higher probability of repeatedly accessing the same subset of experts more frequently.

It’s called mixture of experts but it’s not that concepts map cleanly or even roughly to different experts. Otherwise you wouldn’t get a new expert on every token. You have to remember these were designed to improve throughput in cloud deployments where different GPUs load an expert. There you precisely want each expert to handle randomly to improve your GPU utilization rate. I have not heard anyone training local MoE models to aid sharding.

Re: Unsloth Dynamic 2.0 GGUFs

#58

Post by cagenut »

vlovich123 wrote: post #57
FuckButtons wrote: post #52

Earlier quoted context omitted.

Not entirely true, it’s random access within the relevant subset of experts and since concepts are clustered you actually have a much higher probability of repeatedly accessing the same subset of experts more frequently.

It’s called mixture of experts but it’s not that concepts map cleanly or even roughly to different experts. Otherwise you wouldn’t get a new expert on every token. You have to remember these were designed to improve throughput in cloud deployments where different GPUs load an expert. There you precisely want each expert to handle randomly to improve your GPU utilization rate. I have not heard anyone training local Mo…

is there anywhere good to read/follow to get operational clarity on this stuff?

my current system of looking for 1 in 1000 posts on HN or 1 in 100 on r/locallama is tedious.

Re: Unsloth Dynamic 2.0 GGUFs

#59

Post by gundmc »

hedora wrote: post #51
danielhanchen wrote: post #24

Earlier quoted context omitted.

Oh I didn't expect this to be on HN haha - but yes for our new benchmarks for Qwen3.5, we devised a slightly different approach for quantization which we plan to roll out to all new models from now on!

Nice! Your stuff ran LLMs extremely well on I’m eager to try it out, especially if 16GB is viable now.

The 5080 is 16GB VRAM, not system memory. I don't think you can get 24-32GB VRAM in a $500 box

Re: Unsloth Dynamic 2.0 GGUFs

#60

Post by sowbug »

hnfong wrote: post #42
cpburns2009 wrote: post #39

Earlier quoted context omitted.

Does llama.cpp support Qwen3.5 yet? When I tried it before, it failed saying "qwen35moe" is an unsupported architecture.

Yes, but make sure you grab the latest llama.cpp release New model archs usually involve code changes.

If you're running Ollama, you'll have to wait a little longer for its embedded version of llama.cpp to catch up. It can be a couple days or weeks behind.
Post reply on HN