Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

751–760 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#751

Post by CMay »

beacon294 wrote: post #748
CMay wrote: post #293

Earlier quoted context omitted.

This is my experience too. Qwen optimizes for a lot of scenarios which masks their weaker generalization compared to US frontier models. Never go below an fp16 kv cache unless you've already tested it in advance with your model on a verified task that you know it can successfully complete. People should also test the difference using the exact same seed value so they can see how the tokens diverge. If you have memory…

I find that fp8 cache can be pretty bad in vllm but works fine in llama.cpp. I don't know why, but I plan to review the implementations.

Llama.cpp implemented some rotation optimizations for quantized kv cache to improve the preservation of attention quality or similar, after everyone was talking about TurboQuant. It's not perfect and when you're talking about long form reasoning, little differences can make or break the results so it is situational.

Re: Qwen 3.6 27B is the sweet spot for local development

#752

Post by janalsncm »

hollowturtle wrote: post #630
spiralcoaster wrote: post #427

Earlier quoted context omitted.

[flagged]

These people work mostly in CRUD apps and they're telling you they how feel productive. Btw exploratory ideas even for hard problems come out already after a hackaon of a day or a game jam of 3 days

Do you also believe Terrance Tao was a mediocre mathematician before AI?

Re: Qwen 3.6 27B is the sweet spot for local development

#753

Post by tedivm »

urbsgpw wrote: post #685
tedivm wrote: post #402

Earlier quoted context omitted.

I have a Linux box with two 3090s and it's been great for running Qwen3.6 27b. I lowered the power on each card down to 250w, and then built a small ducting/fan system to vent the waste heat outside. The machine is pretty much silent, and I'm still getting 110 tokens per second out of it for coding tasks. https://github.com/tedivm/qwen36-27b-docker

But is Qwen3.6 27B actually worth this investment? If I had to guess you still use SOTA for architectural/planning work?

No, I use Qwen3.6 27b for everything.

Re: Qwen 3.6 27B is the sweet spot for local development

#754

Post by tedivm »

drnick1 wrote: post #743
tedivm wrote: post #402

Earlier quoted context omitted.

I have a Linux box with two 3090s and it's been great for running Qwen3.6 27b. I lowered the power on each card down to 250w, and then built a small ducting/fan system to vent the waste heat outside. The machine is pretty much silent, and I'm still getting 110 tokens per second out of it for coding tasks. https://github.com/tedivm/qwen36-27b-docker

How useful is the second 3090 in this setup? I run the 5-bit quantized model on a single 3090. Does the second 3090 allow you to use the full precision model instead or a less aggressive quantization by splitting the layers? What about running the 35B model instead?

More memory means less aggressive quantization, more concurrent requests, and larger context windows. I also get a boost in tokens per second (not double, about 1.5x compared to a single GPU).

The 35B model is an MoE (mixture of experts), which uses only a subset of parameters at a time. The 27b one is slower but has way better performance.

Re: Qwen 3.6 27B is the sweet spot for local development

#755

Post by Catloafdev »

rvba wrote: post #635
Catloafdev wrote: post #492

Earlier quoted context omitted.

Q8 is virtually lossless. The quantization is much more noticeable around Q4 and below. FP16->Q8 on consumer hardware is 2x the speed at ~99.99% the quality.

Any source that confirms the 99.99% quality?

I don't have a 'source' off-hand but I recommend reading up on it if you want to learn more. A lot of models on HF show a card demonstrating the different quality trade-offs between quants.

Re: Qwen 3.6 27B is the sweet spot for local development

#756

Post by jboss10 »

CMay wrote: post #335
jboss10 wrote: post #280

I don't understand the talk about how expensive the hardware is. These models can run on very old or old and low end. I've been running Qwen3.6-35B Q4 on an old 1080 GPU(8GB vram) with 32GB sys RAM. I have a i7-12700. It does about 30 tok/s which is enough for me. It's about half what the online models do, but it's enough. I've heard their 9B models are also good, but they aren't much faster if you have the ram and a…

The MoE models hold up better on old hardware, but the dense models like this post promotes are in fact better. This isn't unique to Qwen. Are the dense models better-enough to use given the performance costs? It depends on what you are doing. If a model runs fast enough for your use case and does exactly what you need it to, then you don't need a much slower model that might be more accurate. If you do anything more…

I should try gemma4 more for coding, since qwen3.6 and gemma4 came out I've focused on qwen. For earlier releases I found qwen was smarter, but gemma had more knowledge. But for coding I always want it to learn how to do the task, not just assume/halucinate.

Re: Qwen 3.6 27B is the sweet spot for local development

#757

Post by jboss10 »

TheCycoONE wrote: post #324
jboss10 wrote: post #270

Earlier quoted context omitted.

They can be ran on 32GB with 8GB VRAM. I don't think these will be on 16GB for a while. (35B MoE)

I have 32GB of RAM with 16GB VRAM and I haven't had a lot of luck running larger models like this. Are you able to expand on that?

I'm running llama-swap in a docker container with nvidia container utis to pass through the GPU. This then runs the correct llama-server command to provide the model I want. I have a folder full of guff s I mount in the container.

But this could be done with just llama-server normally. I don't use any special command, just ensure that it's using the GPU. I've found the default fitting to be good.

From memory:

llama-server -m models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -fa on -c 128000

Re: Qwen 3.6 27B is the sweet spot for local development

#758

Post by jboss10 »

XCSme wrote: post #424

Considering the cloud version, all three models compared in the article (Qwen 3.6 35BA3b, 3.6 27B and DeepSeek V4 Flash), have very similar performance[0], BUT on cloud, for some reason DeepSeek V4 Flash is 10-20x cheaper than the Qwen models. If Qwen models are so much easier to run, why are the providers charging more than V4 Flash? [0]: https://aibenchy.com/compare/qwen-qwen3-6-35b-a3b-medium/qwe... <-- compare ho…

Look into deepseek's papers. They have done some stuff recently about improving inference and it seems to be how they can sell tokens so cheap.

Re: Qwen 3.6 27B is the sweet spot for local development

#759

Post by SwellJoe »

mycall wrote: post #614
SwellJoe wrote: post #342

Earlier quoted context omitted.

I opted to buy a normal 32GB laptop for this very reason. I know how loud and hot the GPUs in my desktop run when running even smallish models like Qwen 27B or Gemma 4 31B (which is a better model for most than Qwen 3.6, despite the benchmarks). I also have a Strix Halo which doesn't get loud, because it has a single huge fan, but it does get hot. So, there's no way a laptop could work as hard as models make them wor…

You can limit TDP on Strix Halo so it runs between 32 and 45W which seems to be the sweet spot for heat vs speed.

It's already slow enough. Since it's not loud, I don't really mind. I don't think it's running hot enough to worry about it. It's never become unstable, thus far.

Re: Qwen 3.6 27B is the sweet spot for local development

#760

Post by SwellJoe »

automajicly wrote: post #730
girvo wrote: post #393

Earlier quoted context omitted.

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

I have a M1 Macbook Pro...with only 16gb and I struggled with Qwens2.5-14b trying to do large projects. I loved Qwen but I had to try and do something different. So I switched to Gemma4-12b which looking at it now, seems more like a downgrade than an upgrade.Can you refer me to any Qwen coding models that wont choke my poor 16gb and also connect contextually? I need that context. I love the laser point focus, but I n…

I don't think "large projects" is realistic with a model that fits in ~8GB (I'm assuming you run stuff other than the model). And, Gemma 4 12B QAT at 4-bits is surely the smartest model in its size, but it shines at vision tasks rather than agentic tasks (though it is a good tool user and can do stuff like research, it's obviously not aimed at code).

You can almost always find free models on OpenRouter. Google AI Studio also has free usage of Gemma 4. Both are rate and usage limited which agentic use will probably chew up pretty quick, but you can usually find some pretty powerful models for free. If you rotate through different providers, I think it avoids the cap. Currently several Nemotron models, North Mini Code, Laguna models, Gemma 4 31b and MoE, Qwen 3 Next, and gpt-oss 120b, are all available free on OpenRouter...and better than anything you can run locally in ~8GB.

Post reply on HN