Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

491–500 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#491

Post by [deleted] »

girvo wrote: post #393
SwellJoe wrote: post #342

Earlier quoted context omitted.

I opted to buy a normal 32GB laptop for this very reason. I know how loud and hot the GPUs in my desktop run when running even smallish models like Qwen 27B or Gemma 4 31B (which is a better model for most than Qwen 3.6, despite the benchmarks). I also have a Strix Halo which doesn't get loud, because it has a single huge fan, but it does get hot. So, there's no way a laptop could work as hard as models make them wor…

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

[deleted]

Re: Qwen 3.6 27B is the sweet spot for local development

#492

Post by Catloafdev »

bityard wrote: post #396
Catloafdev wrote: post #69

Earlier quoted context omitted.

Nobody runs unquantized, there's literally no reason to. Q8 would be the largest anyone actually runs on consumer hardware for inference.

Halving the precision of the weights is not a free lunch...

Q8 is virtually lossless. The quantization is much more noticeable around Q4 and below. FP16->Q8 on consumer hardware is 2x the speed at ~99.99% the quality.

Re: Qwen 3.6 27B is the sweet spot for local development

#493

Post by Arch-TK »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

It's okay, completely wrong thread for this statement, but I wouldn't voluntarily use current MacOS (no idea if the older variants weren't terrible) over anything but ssh. Worse than Windows 11.

Re: Qwen 3.6 27B is the sweet spot for local development

#494

Post by SchemaLoad »

mashygpig wrote: post #455

It's fun to run a model locally, but I don't think the economics make sense for anyone just trying to use models atm. It's absurdly cheap to use the same model via openrouter in comparison. Seriously, just put $10 into openrouter and play with models that are cheap but bigger than what you'd reasonably be able to run locally like deepseek v4 flash (unquantized). You'll be surprised by how far that $10 goes for a mode…

Agreed, I'm waiting for the time when 48GB+ ram is just the standard that computers come with rather than being the absolute top tier option. It just doesn't make sense to spend extra on a local AI computer right now when the same money would last for a decade of API pricing.

Re: Qwen 3.6 27B is the sweet spot for local development

#495

Post by aurareturn »

verdverm wrote: post #436
aurareturn wrote: post #385

Earlier quoted context omitted.

273GB/s bandwidth vs 614 GB/s of the M5 Max. And you're getting a whole laptop. $5k for DGX Spark as well.

Prompt processing time is better on the spark, which aligns more with coding (more reading than writing). I spent less than $4k, OEM are better boxes for cooling, no apple markup, I get a real Linux system for stuff like k3s.

Yes, it's better on the Spark but the M5 is a lot closer than before with neural accelrators. After prompt processing, token generation speed on the M5 Max is 2.3x faster.

No Apple markup but you get the Nvidia market up instead. Prior to the recent Apple price increase due to RAM shortage, an M5 Max 128GB was a bargain if you want to run local LLMs.

Re: Qwen 3.6 27B is the sweet spot for local development

#496

Post by pbgcp2026 »

cpburns2009 wrote: post #407
SwellJoe wrote: post #360

Earlier quoted context omitted.

The 8-bit quantized 27B Qwen 3.6 is 29GB. You absolutely cannot run that entirely on a 24GB GPU. You could run a 4-bit, which is 16-17GB. But, you'd need a smallish context or you'd need to quantize your KV cache. Something like TurboQuant or RotorQuant might help. 32GB is the lower bound for comfortably running this size model. I'd maybe even say 64GB is right-sized, because a 256k context is nice to have for agenti…

A 32gb card does run it nicely. I use unsloth's UD-Q5_K_XL at 256k context (k/v at q8_0), and get ~67 t/s on a 5090. I still need to look into MTP.

[dead]

Re: Qwen 3.6 27B is the sweet spot for local development

#498

Post by aand16 »

kroaton wrote: post #480
ljosifov wrote: post #322

Earlier quoted context omitted.

True - they are workhorses. Not super bright, but good enough for lots of everyday tasks. I've found sweet spot to be turning thinking off, as it adds small or no value, while increasing the token count and waiting time. Last 27B I used was https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-GGUF - specifically post-train adapted a bit to run with thinking off. I saw today the 35B-A3B MoE from the same HF acc is out,…

Please don't use that garbage. Just use the base Qwen models or Nex/Orinth, as those are the only properly post-trained finetunes. The Qwopus models are marketing.

Can you expand on why Qwopus is not recommended and what "Nex/Orinth" brings to the table?

Re: Qwen 3.6 27B is the sweet spot for local development

#499

Post by pbgcp2026 »

bityard wrote: post #318
SkitterKherpi wrote: post #254

Earlier quoted context omitted.

You can run 8bit 27B models at 24GB, it's definitely enough for the model size.

Quantization is a trade-off, though. The quality, while still perhaps good enough for many tasks, is not as good as the full 16-bit weights that the model was designed for/released with.

[dead]

Re: Qwen 3.6 27B is the sweet spot for local development

#500

Post by Saris »

mashygpig wrote: post #455

It's fun to run a model locally, but I don't think the economics make sense for anyone just trying to use models atm. It's absurdly cheap to use the same model via openrouter in comparison. Seriously, just put $10 into openrouter and play with models that are cheap but bigger than what you'd reasonably be able to run locally like deepseek v4 flash (unquantized). You'll be surprised by how far that $10 goes for a mode…

Even with deepseek v4 flash I burned though $5 in credits in a day just playing around with Hermes, and qwen 3.6 35B is significantly more expensive.

I can run qwen 3.6 35B on my gaming PC at around 50 tok/s and other than power cost of a tiny bit extra per month, it's hardware I already owned from years ago.

I'm not really sure why qwen 3.6 35B is so expensive on openrouter, it seems abnormally high for what hardware it takes to run it.

Post reply on HN