A bit skeptical about a 27B model comparable to opus...
Opus 4.5 mind you, but I’m not too surprised given how good 3.5 was and how good the qwopus fine tune was. The model was shown to benefit heavily from further RL.
Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
41–50 of 482 posts
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#42I've been waiting for this one. I've been using 3.5-27b with pretty good success for coding in C,C++ and Verilog. It's definitely helped in the light of less Claude availability on the Pro plan now. If their benchmarks are right then the improvement over 3.5 should mean I'm going to be using Claude even less.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#43I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.
> As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless
The 4-bit quants are far from lossless. The effects show up more on longer context problems.
> You can probably even go FP8 with 5090 (though there will be tradeoffs)
You cannot run these models at 8-bit on a 32GB card because you need space for context. Typically it would be Q5 on a 32GB card to fit context lengths needed for anything other than short answers.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#44I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
At 8-bit quantization (q8_0) I get 20 tokens per second on a Radeon R9700.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#45Earlier quoted context omitted.
As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.
That seems awfully speculative without at least some anecdata to back it up.
Not at all, I actually run ~30B dense models for production and have tested out 5090/3090 for that. There are gotchas of course, but the speed/quality claims should be roughly there.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#46I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
I get ~5 tokens/s on an M4 with 32G of RAM, using: llama-server \ -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \ --no-mmproj \ --fit on \ -np 1 \ -c 65536 \ --cache-ram 4096 -ctxcp 2 \ --jinja \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --reasoning on \ --chat-template-kwargs '{"preserve_thinking": true}' 35B-A3B model is at ~25 t/s. For comparison, on an A100 (~RTX…
Why use --fit on on an M4? My understanding was that given the unified memory, you should push all layers to the GPU with --n-gpu-layers all. Setting --flash-attn on and --no-mmap may also get you better results.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#47Earlier quoted context omitted.
As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.
That seems awfully speculative without at least some anecdata to back it up.
Sure, go get some.
This isn't the first open-weight LLM to be released. People tend to get a feel for this stuff over time.
Let me give you some more baseless speculation: Based on the quality of the 3.5 27B and the 3.6 35B models, this model is going to absolutely crush it.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#48A bit skeptical about a 27B model comparable to opus...
A small model can be made to be "comparable to Opus" in some narrow domains, and that's what they've done here.
But when actually employed to write code they will fall over when they leave that specific domain.
Basically they might have skill but lack wisdom. Certainly at this size they will lack anywhere close to the same contextual knowledge.
Still these things could be useful in the context of more specialized tooling, or in a harness that heavily prompts in the right direction, or as a subagent for a "wiser" larger model that directs all the planning and reviews results.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#49I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.
4-bit quantization is almost never lossless especially for agentic work, it's the lowest end of what's reasonable. It's advocated as preferable to a model with fewer parameters that's been quantized with more precision.
Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
#50I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
For Qwen3.5-27b I'm getting in the 20 to 25 tok/sec range on a 128GB Strix Halo box (Framework Desktop). That's with the 8-bit quant. It's definitely usable, but sometimes you're waiting a bit, though I'm not finding it problematic for the most part. I can run the Qwen3-coder-next (80b MoE) at 36tok/sec - hoping they release a Qwen3.6-coder soon.