Unsloth Dynamic 3.0 GGUFs • Hacker News Board

Re: Unsloth Dynamic 3.0 GGUFs

#121

Post by jacquesm » Thu, Aug 20, 2026, 6:51 PM UTC

Alephinitesimal wrote: post #120

jacquesm wrote: post #101

Earlier quoted context omitted.

I'm having a really hard time doing on twin DGX spark what I could do on my quad 3090 rig (which is a scaled down version of what I was using before, the power requirements and the noise were really an issue but I loved the speed and the amount of VRAM). The results tend to be inconsistent, there is lots of looping, far more tokens generated for the same job and lower quality output. I suspect there is some kind of r…

I was only using a single DGX Spark, and this was earlier in the year, so I was running some pretty aggressively quantized models — probably in the 1–3 bit range. My main issue at the time was that my financial data had lots of messy notes, comments, and irregular annotations. The quantized models often failed to process all of that context consistently and would miss things. So I ended up generating a fake dataset w…

Thank you, interesting info! I think the Sparks are an interesting platform, the power consumption / memory bandwidth / memory amount trade-off is completely different from the regular cards and it will take a while for the software to really take advantage of them.

Re: Unsloth Dynamic 3.0 GGUFs

#122

Post by Forgeties79 » Thu, Aug 20, 2026, 11:08 PM UTC

julianlam wrote: post #110

Forgeties79 wrote: post #80

Earlier quoted context omitted.

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster

You're likely using 3.6-35B-A3B, the 3.8 is currently a 27 billion parameter dense model.

I have definitely use that to great success, and I do think it colors some of my memory here. I need to check if the 3.6 27B I was using previously was also a dense model. Good suggestion appreciate it

Re: Unsloth Dynamic 3.0 GGUFs

#123

Post by latentsea » Fri, Aug 21, 2026, 12:36 AM UTC

Forgeties79 wrote: post #80

latentsea wrote: post #73

Earlier quoted context omitted.

Qwen3.8-27B has been the turning point for me. It's not as strong as the absolute frontier, but it's the first time I feel local coding models are actually functionally useable as daily drivers.

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster

Maybe you're holding it wrong because it's the same architecture between the models, assuming you're using the dense 27B model in both cases. And 3.8 is a significant improvement on 3.6.

Re: Unsloth Dynamic 3.0 GGUFs

#124

Post by [deleted] » Fri, Aug 21, 2026, 4:27 AM UTC

jantse wrote: post #105

m1keil wrote: post #104

Earlier quoted context omitted.

I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options. So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8. Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.

Thanks for sharing! Did you observe a speed difference between ollamas mlx version and the mlx-community/Qwen3.8-27B-4bit from HF ran with mlx_vlm.generate (with MTP)? Or is it the same?

[deleted]

Re: Unsloth Dynamic 3.0 GGUFs

#125

Post by m1keil » Fri, Aug 21, 2026, 4:31 AM UTC

jantse wrote: post #105

m1keil wrote: post #104

Earlier quoted context omitted.

I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options. So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8. Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.

Thanks for sharing! Did you observe a speed difference between ollamas mlx version and the mlx-community/Qwen3.8-27B-4bit from HF ran with mlx_vlm.generate (with MTP)? Or is it the same?

huh.. I'm a bit of local LLM noob so I wasn't familiar with mlx_vlm.

I gave it a shot now:

mlx_vlm.generate --model mlx-community/Qwen3.8-27B-4bit --prompt 'give me fizz buzz in rust' --enable-thinking --draft-kind mtp --draft-model mlx-community/Qwen3.8-27B-MTP-4bit --verbose

==========

Prompt: 58 tokens, 90.717 tokens-per-sec Generation: 145 tokens, 36.392 tokens-per-sec Peak memory: 17.419 GB Speculative decoding: 2.79 accepted tokens/round (1.79 accepted drafts/round, 89.4% of drafted, avg draft 2.00) over 52 rounds

Which is very close to ollama, thank you!

I'm not sure if I can get rid of the drafter model, if I understand correctly, the Qwen model already includes a built in draft headers, but just having --draft-kind mtp results in about 17 t/s.