Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 3.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

1–10 of 125 posts

Re: Unsloth Dynamic 3.0 GGUFs

#2

Post by xlayn »

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.

Re: Unsloth Dynamic 3.0 GGUFs

#4

Post by mike-the-brain »

xlayn wrote: post #2

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? impro…

you can still have it, no?

> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed

Re: Unsloth Dynamic 3.0 GGUFs

#5

Post by throwa356262 » 

   "We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller"
This is crazy! But has anyone tried these lower quants on real projects?

Re: Unsloth Dynamic 3.0 GGUFs

#7

Post by kennywinker »

throwa356262 wrote: post #5

"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller" This is crazy! But has anyone tried these lower quants on real projects?

Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts

Re: Unsloth Dynamic 3.0 GGUFs

#9

Post by kamranjon »

jadbox wrote: post #6

The new IQ4XS has been working pretty well so far on 4090 16gb.

What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.
Post reply on HN