Can this help tiny models like Qwen3.5-0.8B?
Unsloth Dynamic 3.0 GGUFs
21–30 of 125 posts
Re: Unsloth Dynamic 3.0 GGUFs
#22Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.
I could of course download and test myself, but that would take days with my internet connection.
Re: Unsloth Dynamic 3.0 GGUFs
#23Earlier quoted context omitted.
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
You don’t even need electricity. You could print the model weights onto millions of sheets of paper, and hire a team of carrier pigeons to fly them into your office one by one. No VRAM!
The bitrot would be excessive
Re: Unsloth Dynamic 3.0 GGUFs
#24Earlier quoted context omitted.
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
You don’t even need electricity. You could print the model weights onto millions of sheets of paper, and hire a team of carrier pigeons to fly them into your office one by one. No VRAM!
IP over Avian Carriers: https://www.rfc-editor.org/info/rfc2549/
Re: Unsloth Dynamic 3.0 GGUFs
#25Cool. Now run TerminalHard and compare to unquantized 27B. KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%
That's not how that works. Selecting a different token is not inherently erroneous. A correct solution can still be found despite divergence.
Re: Unsloth Dynamic 3.0 GGUFs
#26Sidebar: single threaded inference isn’t good enough anymore
What about do you mean by single threaded? Each token is predicted by using parallel computation on the GPU.
Re: Unsloth Dynamic 3.0 GGUFs
#27Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? impro…
The reason for running those insanely low quants is to fit in extremely limited memory budgets. The first thing you sacrifice is speed, then context and accuracy (up to you in which order). IQ2_XXS and below is desperate/proof of concept territory. If you have a spare half gig for the MTP drafter, run a larger quant instead, it will be less incoherent, and damn the speed, it won't be garbage at least. Only around Q4 I'd allocate the comparative luxury of more memory for a speed increase. At least on a dense model. MTP makes a lot more sense (but helps statistically a bit less) on an MoE.
Qwaiting for that 3.8-35B-A3B
Re: Unsloth Dynamic 3.0 GGUFs
#28Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time. I could of course download and test myself, but that would take days with my internet connection.
I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer.
I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.
After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.
Q4_K_P
Re: Unsloth Dynamic 3.0 GGUFs
#29Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
Yes, and if you have the PCIe lanes (say, an x16 lane - actually delivering 16 lanes! - to each GPU) it's also quite performant - it's called a tensor split in llama-server.
If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race.
(Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split)
Edit: If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.
Re: Unsloth Dynamic 3.0 GGUFs
#30Cool. Now run TerminalHard and compare to unquantized 27B. KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%
I don't think you can extrapolate that measurement across multiple sequential draws like that. We presumably are comparing against a single trajectory rather than a tree of trajectories. So once we make the wrong choice and step off of the blessed path, we have no way to assign a ranking to the next token; it's error is undefined.
I've seen LLMs self correct in chains of thought ("because of foo and bar, I need to... Wait, bar is not true, so that won't work") so I have to imagine this is a massive overestimate, errors do not necessarily compound.