Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsloth Dynamic 3.0 GGUFs

unsloth.ai

Post reply on HN
Print view View original

11–20 of 125 posts

Re: Unsloth Dynamic 3.0 GGUFs

#11

Post by xlayn »

mike-the-brain wrote: post #4
xlayn wrote: post #2

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? impro…

you can still have it, no? > We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed

my bad, you are totally right, thanks!

Re: Unsloth Dynamic 3.0 GGUFs

#14

Post by smcleod »

mike-the-brain wrote: post #3

Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?

Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.

Re: Unsloth Dynamic 3.0 GGUFs

#15

Post by charcircuit »

QuantumNomad_ wrote: post #13

Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?

Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.

Re: Unsloth Dynamic 3.0 GGUFs

#18

Post by xlayn »

QuantumNomad_ wrote: post #13

Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?

I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k.

llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1

if you have an igpu and want to exclude or just use some gpus you can use

--device Vulkan3,Vulkan2,Vulkan1

in my case vulkan because of amd, you can see your devices with

llama-server2 --list-devices

Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)

Re: Unsloth Dynamic 3.0 GGUFs

#20

Post by leoooodias »

charcircuit wrote: post #15
QuantumNomad_ wrote: post #13

Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?

Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.

You don’t even need electricity. You could print the model weights onto millions of sheets of paper, and hire a team of carrier pigeons to fly them into your office one by one. No VRAM!
Post reply on HN