Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 3.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

71–80 of 125 posts

Re: Unsloth Dynamic 3.0 GGUFs

#71

Post by zenoprax »

walrus01 wrote: post #47
xlayn wrote: post #2

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? impro…

Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.

Sure, but this is true for all lossy compression (audio, images, etc.)

Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3

Re: Unsloth Dynamic 3.0 GGUFs

#72

Post by zenoprax »

zdragnar wrote: post #54
johndough wrote: post #22

Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time. I could of course download and test myself, but that would take days with my internet connection.

Purely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait...…

I've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.

Re: Unsloth Dynamic 3.0 GGUFs

#73

Post by latentsea »

Alephinitesimal wrote: post #34

I mostly use local models when the data has personal information. Earlier this year, I felt the coding quality was still not as good as Claude Code. One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data. This way the real data never leaves my machine, but I can still use…

Qwen3.8-27B has been the turning point for me. It's not as strong as the absolute frontier, but it's the first time I feel local coding models are actually functionally useable as daily drivers.

Re: Unsloth Dynamic 3.0 GGUFs

#74

Post by zenoprax »

sharmajai wrote: post #17
QuantumNomad_ wrote: post #13

Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?

I am getting 14 t/s on my 16 GB card at full context with the UD-Q3_K_XL quant. Model link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF .

For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.

Re: Unsloth Dynamic 3.0 GGUFs

#75

Post by hedora »

prima-facie wrote: post #65
walrus01 wrote: post #50

It would be nice if unsloth published GGUFs would use a version number or something, because now I have multiple different files on local storage that otherwise have exactly the same name. "Qwen3.8-27B-UD-Q8_K_XL.gguf" for instance. The one downloaded at least 4 days ago is a different thing and is NOT the "Dynamic 3.0" GGUF which I am now downloading, which I presume will have a different sha256 checksum? The unslot…

You can manage these easily with the huggingface python app. hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf and then see them with `hf cache ls`. Prune old versions with `hf cache prune`.

That's terrible. My computer shipped with a filesystem and curl.

Re: Unsloth Dynamic 3.0 GGUFs

#76

Post by mycall »

hedora wrote: post #75
prima-facie wrote: post #65

Earlier quoted context omitted.

You can manage these easily with the huggingface python app. hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf and then see them with `hf cache ls`. Prune old versions with `hf cache prune`.

That's terrible. My computer shipped with a filesystem and curl.

There is always SHA256 as the alternative.

Re: Unsloth Dynamic 3.0 GGUFs

#77

Post by nagaiaida »

gunalx wrote: post #64
walrus01 wrote: post #63

Earlier quoted context omitted.

there's metadata in a GGUF (as can be seen when loading one into llama-server with verbose level 4) but it doesn't appear to show what the expected sha256 of the file is.

Well that would not work as the metadata is also part of the sha, and if you have a sha predictably contain or hash to itself you have a bad hashing algorithm. (Unless of course you mean sha of the weigths and not the gguf itself)

you can just hash the metadata masking out the hash as all zeroes or anything similar to get around that, like header checksums that ostensibly include themselves

Re: Unsloth Dynamic 3.0 GGUFs

#78

Post by stkdump »

zenoprax wrote: post #72
zdragnar wrote: post #54

Earlier quoted context omitted.

Purely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait...…

I've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.

There is a native reasoning effort setting. It defaults to xhigh, I guess to get the best benchmark results, but you can just run it on medium or low instead, or for simple things even disable thinking outright.

Re: Unsloth Dynamic 3.0 GGUFs

#79

Post by illusive4080 »

Alephinitesimal wrote: post #34

I mostly use local models when the data has personal information. Earlier this year, I felt the coding quality was still not as good as Claude Code. One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data. This way the real data never leaves my machine, but I can still use…

Have you tried on personal finance analysis? That is what I most want to do but haven’t gotten around to it.

Re: Unsloth Dynamic 3.0 GGUFs

#80

Post by Forgeties79 »

latentsea wrote: post #73
Alephinitesimal wrote: post #34

I mostly use local models when the data has personal information. Earlier this year, I felt the coding quality was still not as good as Claude Code. One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data. This way the real data never leaves my machine, but I can still use…

Qwen3.8-27B has been the turning point for me. It's not as strong as the absolute frontier, but it's the first time I feel local coding models are actually functionally useable as daily drivers.

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster
Post reply on HN