Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

Post by khimaros » Wed, Apr 22, 2026, 5:40 PM UTC

ryandrake wrote: post #105

Aurornis wrote: post #80

Earlier quoted context omitted.

> Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no problem, but which one do you choose? Unsloth offers[1] four of them: UD-IQ3_XXS, Q3_K_S, Q3_K_M, UD-Q3_K_XL There are actually two problems with this: First, the 3-bit quants are where the quality loss really becomes obvious. You can get it to run, but you’re not getting the quality you expected. The e…

Yea, I'm also kind of jealous of Apple folks with their unified RAM. On a traditional homelab setup with gobs of system RAM and a GPU with relatively little VRAM, all that system RAM sits there useless for running LLMs.

Post by [deleted] » Wed, Apr 22, 2026, 5:44 PM UTC

ryandrake wrote: post #105

Aurornis wrote: post #80

Earlier quoted context omitted.

> Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no problem, but which one do you choose? Unsloth offers[1] four of them: UD-IQ3_XXS, Q3_K_S, Q3_K_M, UD-Q3_K_XL There are actually two problems with this: First, the 3-bit quants are where the quality loss really becomes obvious. You can get it to run, but you’re not getting the quality you expected. The e…

Yea, I'm also kind of jealous of Apple folks with their unified RAM. On a traditional homelab setup with gobs of system RAM and a GPU with relatively little VRAM, all that system RAM sits there useless for running LLMs.

Post by MrDrMcCoy » Wed, Apr 22, 2026, 5:47 PM UTC

amstan wrote: post #100

GrinningFool wrote: post #77

Earlier quoted context omitted.

128GB (112 GB avail) Strix AI 395+ Radeon 8060x (gfx1151) llama-* version 8889 w/ rocm support ; nightly rocm llama.cpp/build/bin/llama-batched-bench --version unsloth/Qwen3.6-27B-GGUF:UD-Q8_K_XL -npp 1000,2000,4000,8000,16000,32000 -ntg 128 -npl 1 -c 34000 | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|-------…

you should try vulkan instead of rocm. it goes like 20% faster.

Post by [deleted] » Wed, Apr 22, 2026, 5:49 PM UTC

syntaxing wrote: post #136

acidtechno303 wrote: post #122

Earlier quoted context omitted.

Sounds like you're accusing a professional of holding their tool incorrectly. Not impossible, but not likely either.

Inferencing is straight up hard. I’m not accusing them of anything. There’s a crap ton of variables that can go into running a local model. No one runs them at native FP8/FP16 because we cannot afford to. Sometimes llama cpp implementation has a bug (happens all the time). Sometimes the template is wrong. Sometimes the user forgot to expand the context length to above the 4096 default. Sometimes they use quantization…

Post by anonym29 » Wed, Apr 22, 2026, 5:49 PM UTC

p_stuart82 wrote: post #159

anonym29 wrote: post #117

Earlier quoted context omitted.

You absolutely do not need to run at full BF16. The quality loss between BF16 (55.65 GB in GGUF) and Q8_0 (30.44 GB in GGUF) is essentially zero - think on the order of magnitude of +0.01-0.03 perplexity, or ~0.1-0.3% relative PPL increase. The quality loss between BF16 and Q4_K_M (18.66 GB in GGUF) is close to imperceptible, with perplexity changes in the +0.1-0.3 ballpark, or ~1-3% relative PPL increase. This would…

tbh ~1-3% PPL hit from Q4_K_M stopped being the bottleneck a while ago. the bottleneck is the 48 hours of guessing llama.cpp flags and chat template bugs before the ecosystem catches up. you are doing unpaid QA.