11–20 of 141 posts
There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.
Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work th…
> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.
There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.
> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…
/data/llm/llama.cpp/build/bin/llama-server
--threads 4
--threads-batch 8
--batch-size 4096
--ubatch-size 256
--port 9999
--temp "1.0"
--top-p "0.95"
--top-k "20"
--min-p "0.0"
--presence-penalty "0.0"
--reasoning auto
--reasoning-preserve
--reasoning-budget 4096
--gpu-layers-draft all
--spec-type draft-mtp,ngram-map-k4v,ngram-mod
--spec-draft-n-max 3
--spec-draft-p-min 0.75
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 4
--spec-ngram-mod-n-max 16
--spec-ngram-map-k4v-size-n 8
--spec-ngram-map-k4v-size-m 16
--spec-ngram-map-k4v-min-hits 1
--n-gpu-layers all
--ctx-size 131072
--repeat-penalty 1.0
--jinja
--metrics
--model /data/llm/models/unsloth/Qwen3.8-27B-UD-IQ3_S.gguf
--chat-template-file /data/llm/models/qwen3.6-chat-template.jinja
--fit off
--flash-attn on
--cors-origins localhost
--mmproj /data/llm/models/unsloth/Qwen3.8/mmproj-BF16.gguf
--no-mmproj-offload
--parallel 1
--kv-unified
--cache-type-k q4_0
--cache-type-v q4_0
--cache-type-k-draft q4_0
--cache-type-v-draft q4_0There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.
Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!
Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work th…