Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

qwen.ai

Post reply on HN
Print view View original

181–190 of 482 posts

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#181

Post by zargon »

Aurornis wrote: post #80
ryandrake wrote: post #67

Earlier quoted context omitted.

Yea, this is currently the confusing part of running local models for newbies: Even after you have decided which model you want to run, and which org's quantizations to use (let's just assume Unsloth's for example), there are often dozens of quantizations offered, and choosing among them is confusing. Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no pro…

> Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no problem, but which one do you choose? Unsloth offers[1] four of them: UD-IQ3_XXS, Q3_K_S, Q3_K_M, UD-Q3_K_XL There are actually two problems with this: First, the 3-bit quants are where the quality loss really becomes obvious. You can get it to run, but you’re not getting the quality you expected. The e…

Qwen3.5 series is a little bit of an exception to the general rule here. It is incredibly kv cache size efficient. I think the max context (262k) fits in 3GB at q8 iirc. I prefer to keep the cache at full precision though.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#182

Post by lambda »

UncleOxidant wrote: post #50
anonzzzies wrote: post #11

I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.

For Qwen3.5-27b I'm getting in the 20 to 25 tok/sec range on a 128GB Strix Halo box (Framework Desktop). That's with the 8-bit quant. It's definitely usable, but sometimes you're waiting a bit, though I'm not finding it problematic for the most part. I can run the Qwen3-coder-next (80b MoE) at 36tok/sec - hoping they release a Qwen3.6-coder soon.

That sounds high for a Strix Halo with a dense 27b model. Are you talking about decode (prompt eval, which can happend in parallel) or generation when you quote tokens per second? Usually if people quote only one number they're quoting generation speed, and I would be surprised if you got that for generation speed on a Strix Halo.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#183

Post by acrispino »

ndriscoll wrote: post #70
Aurornis wrote: post #23

Earlier quoted context omitted.

The 27B model they release directly would require significant hardware to run natively at 16-bit: A Mac or Strix Halo 128GB system, multiple high memory consumer GPUs, or an RTX 6000 workstation card. This is why they don’t advertise which consumer hardware it can run on: Their direct release that delivers these results cannot fit on your average consumer system. Most consumers don’t run the model they release direct…

Note that you could also run them on AMD (and presumably Intel) dGPUs. e.g. I have a 32GB R9700, which is much cheaper than a 5090, and runs 27B dense models at ~20 t/s (or MoE models with 3-4B active at ~80t/s). I expect an Arc B70 would also work soon if it doesn't already, and would likely be the price/perf sweet spot right now. My R9700 does seem to have an annoying firmware or driver bug[0] that causes the fan t…

I have 2x asrock R9700. One of the them was noticeably noisier than the other and eventually developed an annoying vibration while in the middle of its fan curve. Asrock replaced it under RMA.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#184

Post by yougotwill »

mark_l_watson wrote: post #151

I have been running the slightly larger 31B model for local coding: ollama launch claude --model qwen3.6:35b-a3b-nvfp4 This has been optimized for Apple Silicon and runs well on a 32G ram system. Local models are getting better!

Can I ask how much RAM of the 32GB does it use? For example can I run a browser and VS Code at the same time?

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#185

Post by alex7o »

Aurornis wrote: post #43
ekojs wrote: post #32

Earlier quoted context omitted.

As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.

> As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless The 4-bit quants are far from lossless. The effects show up more on longer context problems. > You can probably even go FP8 with 5090 (though there will be tradeoffs) You cannot run these models at 8-bit on a 32GB card because you need space for context. Typically it would be Q5 on a 32GB card to fit context lengths needed f…

Turboquant on 4bit helps a lot as well for keeping context in vram, but int4 is definitely not lossless. But it all depends for some people this is sufficient

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#186

Post by simonw »

refulgentis wrote: post #155
taspeotis wrote: post #145

Earlier quoted context omitted.

Why would you though? And by the way: Thanks for relentlessly holding new models’ feet to the pelican SVG fire.

Because I want to read about Qwen, not someone's one-off vibe test followed by 1:1 conversations. (case in miniature here: which is the last comment in this thread that says something about Qwen? The root post. Is that fun policing? Yes, apologies.)

There's a bunch of useful information in my comment that's independent of the fact that it drew a pelican:

1. You can run this on a Mac using llama-server and a 17GB downloaded file

2. That version does indeed produce output (for one specific task) that's of a good enough quality to be worth spending more time checking out this model

3. It generated 4,444 tokens in 2min 53s, which is 25.57 tokens/s

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#187

Post by simonw »

ahoog42 wrote: post #174
simonw wrote: post #106

The pelican is excellent for a 16.8GB quantized local model: https://simonwillison.net/2026/Apr/22/qwen36-27b/ I ran it on an M5 Pro with 128GB of RAM, but it only needs ~20GB of that. I expect it will run OK on a 32GB machine. Performance numbers: Reading: 20 tokens, 0.4s, 54.32 tokens/s Generation: 4,444 tokens, 2min 53s, 25.57 tokens/s I like it better than the pelican I got from Opus 4.7 the other day: https://si…

at what point do model providers optimize for the "pelican riding a bicycle" test so they place well on Simon's influential benchmark? :-)

See https://simonwillison.net/2025/Nov/13/training-for-pelicans-...

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#188

Post by Abby_101 »

butz wrote: post #142

Are there any "optimized" models, that have lesser hardware requirements and are specialised in single programming language, e.g. C# ?

Sort of - there's Qwen3-Coder and the Codestral family, but those are still multi-language, just code-focused. For truly single-language specialization, the practical path is fine-tuning an existing base model on a narrow distribution rather than training from scratch.

The issue with C# specifically is dataset availability. Open source C# code on GitHub is a fraction of Python/JS, and Microsoft hasn't released a public corpus the way Meta has for their code models. You'd probably get further fine-tuning Qwen3-Coder (or a similar base) on your specific codebase with LoRA than waiting for a dedicated C#-only model to appear.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#189

Post by smallerize »

Aurornis wrote: post #80
ryandrake wrote: post #67

Earlier quoted context omitted.

Yea, this is currently the confusing part of running local models for newbies: Even after you have decided which model you want to run, and which org's quantizations to use (let's just assume Unsloth's for example), there are often dozens of quantizations offered, and choosing among them is confusing. Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no pro…

> Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no problem, but which one do you choose? Unsloth offers[1] four of them: UD-IQ3_XXS, Q3_K_S, Q3_K_M, UD-Q3_K_XL There are actually two problems with this: First, the 3-bit quants are where the quality loss really becomes obvious. You can get it to run, but you’re not getting the quality you expected. The e…

I found the KLD benchmark image at the bottom of https://unsloth.ai/docs/models/qwen3.6 to be very helpful when choosing a quant.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#190

Post by adrian_b »

anonym29 wrote: post #139
zozbot234 wrote: post #126

Earlier quoted context omitted.

I'm all for running large MoE models on unified memory systems, but developers of inference engines should do a better job of figuring out how to run larger-than-total-RAM models on such systems, streaming in sparse weights from SSD but leveraging the large unified memory as cache. This is easily supported with pure-CPU inference via mmap, but there is no obvious equivalent when using the GPU for inference.

I use llama.cpp, and there is a way to do this - some layers to (i)GPU, the rest to CPU. I was just trying this out with Kimi K2.5 (in preparation for trying it out with Kimi K2.6 the other night. Check out the --n-cpu-moe flag in llama.cpp. That said, my Strix Halo rig only has PCIe 4.0 for my NVMe, and I'm using a 990 Evo that had poor sustained random read, being DRAM-less. My effective read speeds from disk were…

In a computer with 2 PCIe 5.0 SSDs or one with a PCIe 5.0 SSDs and a PCIe 4.0 SSD, it should be possible to stream weights from the SSDs at 20 GB/s, or even more.

This is not a little faster, but 10 times faster than on your system. So a couple of tokens per second generation speed should be achievable.

Nowadays even many NUCs or NUC-like mini-PCs have such SSD slots.

I have actually started working at optimizing such an inference system, so your data is helpful for comparison.

Post reply on HN