Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
Yes, google "llama-server split model multiple GPU" for some concrete examples.
Unsloth Dynamic 3.0 GGUFs
51–60 of 125 posts
Re: Unsloth Dynamic 3.0 GGUFs
#52Would converting those quant to MLX preserve the accuracy/size ?
Or this only work with GGUF?
Re: Unsloth Dynamic 3.0 GGUFs
#53Since it seems like this not only improved sizes but also performance I can't wait for some benchmarks and comparisons. If you don't have a separate GPU for inference, every single GB matters so a comparison between specific Q4 Quants is really interesting to me.
Currently I very much can't decide between going for a bit of a lower Q4 Quant to squeeze out a bit of buffer and ctx or wondering if a slightly higher (IQ4_XS vs Q4_K_M/XL) is worth it
Re: Unsloth Dynamic 3.0 GGUFs
#54Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time. I could of course download and test myself, but that would take days with my internet connection.
Purely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait...
It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.
Re: Unsloth Dynamic 3.0 GGUFs
#55[flagged]
Re: Unsloth Dynamic 3.0 GGUFs
#56Earlier quoted context omitted.
I don't think you can extrapolate that measurement across multiple sequential draws like that. We presumably are comparing against a single trajectory rather than a tree of trajectories. So once we make the wrong choice and step off of the blessed path, we have no way to assign a ranking to the next token; it's error is undefined. I've seen LLMs self correct in chains of thought ("because of foo and bar, I need to...…
actually, bar is true but wait, the models constantly go back and forth on these things in their thinking traces, so it is unclear which self correcting is actually correct
It doesn't really matter to this argument, if they are self correcting at all, then we can't assume all errors will permanently injure the trajectory. It's not like dead reckoning or a similar process where there is never an opportunity to reassess. It's more like a long division problem; it is possible to correct errors using in band information, without the external reference dead reckoning would require. (The incidence of false positive self correction does matter to the question of whether the model is actually of comparable quality after the quantization, of course.)
Re: Unsloth Dynamic 3.0 GGUFs
#57"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller" This is crazy! But has anyone tried these lower quants on real projects?
I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly. If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.
What setup are you using to do said private evaluation? Software wise I mean
Re: Unsloth Dynamic 3.0 GGUFs
#58No Dynamic 3.0 NVFP4 quants just yet from the look of it, as a heads up. Would love to see how those perform relative to others on the curve.
Re: Unsloth Dynamic 3.0 GGUFs
#59Earlier quoted context omitted.
I don't think you can extrapolate that measurement across multiple sequential draws like that. We presumably are comparing against a single trajectory rather than a tree of trajectories. So once we make the wrong choice and step off of the blessed path, we have no way to assign a ranking to the next token; it's error is undefined. I've seen LLMs self correct in chains of thought ("because of foo and bar, I need to...…
I would say they do compound until proven otherwise. Having "Wait, bar is not true, so that won't work" is not necessarily a correction. In fact, the problem is: across a long text it is a correction of a single mistake, but we are talking about thousands here. But yes, of course that was a rough estimate. But the problem is - we don't really know what we are measuring here. Maybe there's a 2,000,000x difference of i…
Just at a sniff test level, don't you think that if the quantization resulted in anything like 2M% error in a pretty typical context length, it would be plain as day? You'd do an A/B test and one of them would look like standard generated text and one of them would veer into incoherence? If not - what on would 2M% error even mean then?
> Regardless the point is KLD and whatever they came up with is not meaningful.
I'm not saying you're wrong, I'm just saying this isn't a meaningful metric either, mostly because it is using a different type of error (divergence along a trajectory) than what was actually measured (divergence at a fixed point) and so can't be used for this purpose. It could establish an upper bound but going by your work that upper bound is so high it may as well be infinite. That's somewhat concerning but doesn't necessarily suggest it performs badly in a typical case (which is how I'd interpret an expectation of 2M% error).
Re: Unsloth Dynamic 3.0 GGUFs
#60It would be nice if unsloth published GGUFs would use a version number or something, because now I have multiple different files on local storage that otherwise have exactly the same name. "Qwen3.8-27B-UD-Q8_K_XL.gguf" for instance. The one downloaded at least 4 days ago is a different thing and is NOT the "Dynamic 3.0" GGUF which I am now downloading, which I presume will have a different sha256 checksum? The unslot…
How can I tell which one I have? I dl'd mine a few days ago.