Qwen 3.7 Preview

Post by nyrikki » Mon, May 18, 2026, 10:01 PM UTC

nyrikki wrote: post #67

Vaskivo wrote: post #59

Earlier quoted context omitted.

How can you get it to run at 41 t/s? I also have a single 3090 and even with MTP can't break 20 t/s. HEre's my setup: llama-server --port 9999 --model /MODELS/LLMs/Qwen3.6-27B-UD-Q4_K_XL.gguf --ctx-size 128000 --threads 12 --flash-attn on --device CUDA0 --jinja --gpu-layers 52 --mmproj /MODELS/LLMs/Qwen3.6-27B-mmproj-F16.gguf --cache-type-k q8_0 --cache-type-v q8_0 --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 --rep…

(Note UPDATED config) Ya, if you are using the CPU it may slowdown quick. This may be a bit huge and overcomplicated, on this host I am running it on a AMD Ryzen 7 5700G so that I can use the APU to dedicate the 3090. podman run --device nvidia.com/gpu=all -d -v llama_qwen3.6mpt:/root/.cache -p 8080:8080 local/llama.cpp:full-cuda --server \ -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL \ -ngl 99 \ --ctx-size 131072 \ -…

     podman run --device nvidia.com/gpu=all -d -v llama_qwen3.6mpt:/root/.cache -p 8080:8080 local/llama.cpp:full-cuda --server \
    -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL \
    -ngl 99 \
    --ctx-size 128000 \
    --no-mmproj-offload \
    --no-context-shift \
    --kv-unified \
    --spec-type draft-mtp \
    --spec-draft-n-max 2 \
    --spec-draft-p-min 0.75 \
    -fa on --jinja --no-mmap \
    --cache-ram -1 \
    --no-warmup -np 1\
    -n 32768 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --temp 0.6 \
    --min-p 0.00 \
    --top-k 20 \
    --top-p 0.95 \
    --presence-penalty 0.0 \
    --repeat-penalty 1.05 \
    --fit off \
    --reasoning on \
    --chat-template-kwargs '{"preserve_thinking":true}' \
    --port 8080 \
    --host 0.0.0.0

Post by fgonzag » Mon, May 18, 2026, 10:15 PM UTC

genxy wrote: post #37

sleepyeldrazi wrote: post #28

Earlier quoted context omitted.

Finetuning takes little resources, the base model training is the slow and expensive part. Architecturally 3.5 models are identical to their 3.6 counterparts, that is why there is a consensus that those are probably finetunes and not re-trained from scratch, like you will se many people publish their own on huggingface.

Understood, but look at their larger cadence over the years and the breadth of models. They are clearly not all finetunes. Meta for all its billions, doesn't have anything comparable.

Post by julianlam » Mon, May 18, 2026, 10:56 PM UTC

Post by kennywinker » Mon, May 18, 2026, 10:57 PM UTC

SwellJoe wrote: post #52

svachalek wrote: post #29

Earlier quoted context omitted.

To play devil's advocate I do feel like Grok has a unique "feel" to it. All the Chinese models feel like GPT or Claude distillations, but Grok has a certain unique way of saying and doing things. But that said, it also feels a year behind the state of the art.

With an Austrian accent, perhaps?

Post by cyanydeez » Mon, May 18, 2026, 11:17 PM UTC

sleepyeldrazi wrote: post #8

I don't think I can handle another small model release by qwen, I'm still trying to find the limits of 3.6 27B and they are already threatening us with a new one? But jokes aside, I love the fast iteration, these are most probably again finetunes on the 3.5 architecture that appear better in internal testing, which is still very nice to see. Putting more and more pressure on the bigger labs to perform better is alway…