Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 3.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

111–120 of 125 posts

Re: Unsloth Dynamic 3.0 GGUFs

#111

Post by jwr »

m1keil wrote: post #104
jwr wrote: post #103

These are very good! I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.

I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options. So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8. Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.

Hmm, perhaps I should switch to an MLX version… problem is, it took quite a bit of work to get llama-server (with llama.cpp) to serve my model(s) and allow requests in non-thinking (default) and thinking modes.

But 30-40 tokens/s would make a big difference.

Re: Unsloth Dynamic 3.0 GGUFs

#112

Post by idonotknowwhy »

danielhanchen wrote: post #43
xlayn wrote: post #2

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? impro…

Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed. As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that. But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

>But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

For those of us with a 16GB GPU, how do they compare with ExllamaV4 at 4-bit (4.0bpw)?

It looks like that fits in 12.5GB of VRAM since embedding are left in DRAM, Unsloth Studio and other llama.cpp derivatives have to load these weights in VRAM for tied embedding models like Qwen3.8.

ExllamaV3 4.0bpw fits in 12.5G of VRAM and beats IQ4_XS according to the measurements here: [turboderp/Qwen3.8-27B-exl3](https://huggingface.co/turboderp/Qwen3.8-27B-exl3)

But those were compared against UD2.0 I guess. Also plans to support these (SOTA) quants in Unsloth Studio?

Re: Unsloth Dynamic 3.0 GGUFs

#113

Post by idonotknowwhy »

zenoprax wrote: post #71
walrus01 wrote: post #47

Earlier quoted context omitted.

Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.

Sure, but this is true for all lossy compression (audio, images, etc.) Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3

If it's a Nvidia card 3000 series or newer, I'd try 4.0bpw ExllamaV3 if you haven't already. Otherwise it look like UD3.0 Q3_K_XL based on the Unsloth blog post.

Re: Unsloth Dynamic 3.0 GGUFs

#114

Post by prima-facie »

14u2c wrote: post #83
prima-facie wrote: post #65

Earlier quoted context omitted.

You can manage these easily with the huggingface python app. hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf and then see them with `hf cache ls`. Prune old versions with `hf cache prune`.

Using git is another option, you'll get the actual commits/history to reference: GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/unsloth/Qwen3.8-27B-GGUF cd Qwen3.8-27B-GGUF git lfs pull --include="Qwen3.8-27B-UD-Q8_K_XL.gguf" --exclude=""

This is very cool, thanks

Re: Unsloth Dynamic 3.0 GGUFs

#116

Post by zenoprax »

lifepillar wrote: post #88
stkdump wrote: post #78

Earlier quoted context omitted.

There is a native reasoning effort setting. It defaults to xhigh, I guess to get the best benchmark results, but you can just run it on medium or low instead, or for simple things even disable thinking outright.

According to this guy [0], medium is the level that tends to produce way less tokens in agentic workflows ("low" may output less per response, but then the model makes more mistakes, so it needs to iterate more). [0] https://m.youtube.com/watch?v=z64J6bC16iQ

Great, thanks! I'll try it out.

Re: Unsloth Dynamic 3.0 GGUFs

#117

Post by zenoprax »

idonotknowwhy wrote: post #113
zenoprax wrote: post #71

Earlier quoted context omitted.

Sure, but this is true for all lossy compression (audio, images, etc.) Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3

If it's a Nvidia card 3000 series or newer, I'd try 4.0bpw ExllamaV3 if you haven't already. Otherwise it look like UD3.0 Q3_K_XL based on the Unsloth blog post.

I'll try it out, thanks. (Using an AMD 9070 XT)

Re: Unsloth Dynamic 3.0 GGUFs

#118

Post by boredatoms »

14u2c wrote: post #83
prima-facie wrote: post #65

Earlier quoted context omitted.

You can manage these easily with the huggingface python app. hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf and then see them with `hf cache ls`. Prune old versions with `hf cache prune`.

Using git is another option, you'll get the actual commits/history to reference: GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/unsloth/Qwen3.8-27B-GGUF cd Qwen3.8-27B-GGUF git lfs pull --include="Qwen3.8-27B-UD-Q8_K_XL.gguf" --exclude=""

Ah this feels much nicer than learning yet another custom vendor cli thing

Re: Unsloth Dynamic 3.0 GGUFs

#119

Post by kristjansson »

14u2c wrote: post #83
prima-facie wrote: post #65

Earlier quoted context omitted.

You can manage these easily with the huggingface python app. hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf and then see them with `hf cache ls`. Prune old versions with `hf cache prune`.

Using git is another option, you'll get the actual commits/history to reference: GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/unsloth/Qwen3.8-27B-GGUF cd Qwen3.8-27B-GGUF git lfs pull --include="Qwen3.8-27B-UD-Q8_K_XL.gguf" --exclude=""

hf download maintains each downloaded version in a commit-hash-addressed snapshots dir, with symlinks to the underlying files content-addressed in a blobs dir. It’s not quite git, but it is a stable reference path.

Re: Unsloth Dynamic 3.0 GGUFs

#120

Post by Alephinitesimal »

jacquesm wrote: post #101
Alephinitesimal wrote: post #91

Earlier quoted context omitted.

I did try some finance analysis earlier this year. I was using a DGX Spark, so I could run some relatively large models, but the results were pretty mixed at the time. I honestly can't remember which models I used anymore. Might be worth trying again now though.

I'm having a really hard time doing on twin DGX spark what I could do on my quad 3090 rig (which is a scaled down version of what I was using before, the power requirements and the noise were really an issue but I loved the speed and the amount of VRAM). The results tend to be inconsistent, there is lots of looping, far more tokens generated for the same job and lower quality output. I suspect there is some kind of r…

I was only using a single DGX Spark, and this was earlier in the year, so I was running some pretty aggressively quantized models — probably in the 1–3 bit range.

My main issue at the time was that my financial data had lots of messy notes, comments, and irregular annotations. The quantized models often failed to process all of that context consistently and would miss things. So I ended up generating a fake dataset with the same structure, asking Claude Code to work out the analysis on that, and then bringing the result back to the local model for the final pass.

I was mainly using llama.cpp at the time, before B12X support was integrated into vLLM, so I think I wasn't using it then.

Post reply on HN