Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 3.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

81–90 of 125 posts

Re: Unsloth Dynamic 3.0 GGUFs

#81

Post by josh-wrale »

sosodev wrote: post #46
josh-wrale wrote: post #31

Earlier quoted context omitted.

Multiple agents need tokens. Should optimize for that instead of one agent blocking the others.

One agent typically blocks the others on a local device because the GPU is already completely utilized either in terms of memory or compute. You can have true parallelism at home, but you need an absurd amount of resources. It's not a simple threading problem.

Yes, but I haven’t seen it in Unsloth/llama.cpp. I see it in Sglang and vllm. Unsloth should default to sglang imo

Re: Unsloth Dynamic 3.0 GGUFs

#82

Post by kQq9oHeAz6wLLS »

charcircuit wrote: post #15
QuantumNomad_ wrote: post #13

Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?

Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.

The key to running in lower amounts of VRAM is patience. It'll be slow, but it'll work.

Re: Unsloth Dynamic 3.0 GGUFs

#83

Post by 14u2c »

prima-facie wrote: post #65
walrus01 wrote: post #50

It would be nice if unsloth published GGUFs would use a version number or something, because now I have multiple different files on local storage that otherwise have exactly the same name. "Qwen3.8-27B-UD-Q8_K_XL.gguf" for instance. The one downloaded at least 4 days ago is a different thing and is NOT the "Dynamic 3.0" GGUF which I am now downloading, which I presume will have a different sha256 checksum? The unslot…

You can manage these easily with the huggingface python app. hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf and then see them with `hf cache ls`. Prune old versions with `hf cache prune`.

Using git is another option, you'll get the actual commits/history to reference:

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

cd Qwen3.8-27B-GGUF

git lfs pull --include="Qwen3.8-27B-UD-Q8_K_XL.gguf" --exclude=""

Re: Unsloth Dynamic 3.0 GGUFs

#85

Post by kamranjon »

xlayn wrote: post #45
danielhanchen wrote: post #43

Earlier quoted context omitted.

Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed. As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that. But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

Daniel, question I got the Qwen3.8-27B-UD-Q2_K_XL.gguf from https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_in... and continue with my testing, but the model quickly felt into a loop of asking the same thing over and over again, I have seen the MOE do that but not the dense ones. And I had similar experiences when Qwen3.8-27B unsloth images just came out with the full Q8_K_XL, I'm using an AMD setup which h…

Are you using the recommended settings for temperature and such? https://unsloth.ai/docs/models/qwen3.8#recommended-settings

Often times I run into issues like this it’s because I am using settings for a different model or just forget to set them up.

Re: Unsloth Dynamic 3.0 GGUFs

#86

Post by andai »

Aurornis wrote: post #48
throwa356262 wrote: post #5

"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller" This is crazy! But has anyone tried these lower quants on real projects?

I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly. If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.

I wonder if this would help, or if it solves different kinds of errors.

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

https://news.ycombinator.com/item?id=48192383

Re: Unsloth Dynamic 3.0 GGUFs

#88

Post by lifepillar »

stkdump wrote: post #78
zenoprax wrote: post #72

Earlier quoted context omitted.

I've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.

There is a native reasoning effort setting. It defaults to xhigh, I guess to get the best benchmark results, but you can just run it on medium or low instead, or for simple things even disable thinking outright.

According to this guy [0], medium is the level that tends to produce way less tokens in agentic workflows ("low" may output less per response, but then the model makes more mistakes, so it needs to iterate more).

[0] https://m.youtube.com/watch?v=z64J6bC16iQ

Re: Unsloth Dynamic 3.0 GGUFs

#89

Post by walrus01 »

zenoprax wrote: post #71
walrus01 wrote: post #47

Earlier quoted context omitted.

Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.

Sure, but this is true for all lossy compression (audio, images, etc.) Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3

probably the best experience would be deepseek v4 flash 0731 (it takes about 170GB RAM on the server side for the full thing and RAM reserved for 1M context) via opencode's $10 a month plan until you use that up, it's either Q8 or full precision. Assuming you're ok with doing things with external inference.

Re: Unsloth Dynamic 3.0 GGUFs

#90

Post by Alephinitesimal »

Forgeties79 wrote: post #80
latentsea wrote: post #73

Earlier quoted context omitted.

Qwen3.8-27B has been the turning point for me. It's not as strong as the absolute frontier, but it's the first time I feel local coding models are actually functionally useable as daily drivers.

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster

That's interesting since both models are dense. I wonder if this is more of an optimization issue with 3.8 rather than something inherent to the architecture.
Post reply on HN