Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

671–680 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#671

Post by gyanchawdhary »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

This is a very exaggerated take. I have an Apple M5 Max with 128 GB ram running 15'ish Coasts (coasts.dev) environments, each of them running postgress, python, redis and FE stack + locally running voice models and face swap models .. and the only time the fan kicks in is when I open multiple google analytics tabs.

Re: Qwen 3.6 27B is the sweet spot for local development

#672

Post by KingMob »

smt88 wrote: post #181
AlpacaJones wrote: post #150

Earlier quoted context omitted.

The key word there is 'currently'.

Economies of scale are a fact of nature and aren’t going to be subverted in the future by even the most advanced local models

Setting aside that very little about economics rises to the level of "facts of nature" like physics...

What makes you so certain that economies of scale won't work the opposite way you imagine? E.g., if model improvement tapers off, but RAM costs decline (hard to believe atm, but historically likely), then eventually everyone will be able to run SOTA models on their personal hardware.

Heck, even if model sizes simply grow more slowly than RAM costs decrease, the same would happen.

Re: Qwen 3.6 27B is the sweet spot for local development

#673

Post by anon373839 »

nozzlegear wrote: post #188
beastman82 wrote: post #43

FWIW I'm running gemma4 31b on my 5090 and it's pretty great as well. QAT, MTP, 128k context. I liked Qwen 3.6 27b too, it just seems that Gemma4 is a bit underrated.

I can't Gemma4 to actually finish a turn properly, it's always ending abruptly or making malformed tool calls. It's probably something I've misconfigured in oMLX or Opencode.

Same problem with Gemma 4 + oMLX + OpenCode. The thinking and tool calling seems to be parsed fine in other clients such as Open WebUI. This really shouldn’t even matter because the client isn’t responsible for parsing the output, but it’s happening anyway.

Re: Qwen 3.6 27B is the sweet spot for local development

#674

Post by Terretta »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

I have that model, and do local LLMs and local image generation. DO buy this if you plan on serious local LLM use and enjoy working from anywhere.

Don't expect workstation loads with no fan or heatsink, true. But it's not a real problem, it's still quieter than a desktop.

That said, rather than Mac Mini, if you only work from one place, I'd recommend a Studio Ultra M3 with 512GB. Same or more tokens per second, multiple models loaded. Cool and quiet.

Re: Qwen 3.6 27B is the sweet spot for local development

#675

Post by [deleted] »

lee_ars wrote: post #323
acters wrote: post #212

Earlier quoted context omitted.

Would the new upcoming AMD AI ryzen halo desktop be a better value offer? or dgx spark? You would have to get a third party reseller/scalper or refurbished mac mini to get 64gb of ram ever since apple stopped selling it.

I'm currently fiddling with a DGX Spark and Qwen3.6-35B-A3B (specifically Qwen3.6-35B-A3B-NVFP4 under vLLM, with EAGLE3 speculative decoding via eagle3-dogacel-vllm), and it's pretty okay in terms of smarts. The speed is relatively usable at about 50 tok/sec with a 256k context window, and it's definitely smart enough to one-shot some basic coding tasks. I had it doing reverse engineering/disassembly of some ancient…

[deleted]

Re: Qwen 3.6 27B is the sweet spot for local development

#676

Post by myzek »

Saris wrote: post #500
mashygpig wrote: post #455

It's fun to run a model locally, but I don't think the economics make sense for anyone just trying to use models atm. It's absurdly cheap to use the same model via openrouter in comparison. Seriously, just put $10 into openrouter and play with models that are cheap but bigger than what you'd reasonably be able to run locally like deepseek v4 flash (unquantized). You'll be surprised by how far that $10 goes for a mode…

Even with deepseek v4 flash I burned though $5 in credits in a day just playing around with Hermes, and qwen 3.6 35B is significantly more expensive. I can run qwen 3.6 35B on my gaming PC at around 50 tok/s and other than power cost of a tiny bit extra per month, it's hardware I already owned from years ago. I'm not really sure why qwen 3.6 35B is so expensive on openrouter, it seems abnormally high for what hardwar…

How do you run 35B on a gaming PC?

I'm trying to go the same route, but I have a 5070Ti with only 16GB VRAM (I bought it for gaming) and I'm not sure how to run anything decent on it. I have 64 GB RAM if that matters

Re: Qwen 3.6 27B is the sweet spot for local development

#677

Post by SomeHacker44 »

organsnyder wrote: post #66
bensyverson wrote: post #26

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs. [0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

I run Qwen 3.6 on my Framework Desktop 128GB, and it's very performant. I know Framework has had to raise the price since I preordered mine, but they're still well under half the cost of that Macbook.

Can you please explain how you set it up? I run it on my 129G Strix Halo under Arch with Lemonade with OpenCode and it just sits there doing barely anything unless I leave it to run over night. Then it says it thought for 13.7 seconds but was really 15 minutes. Thanks! I am using the 27B dense MTP model quantized by UnSloth with the UD-Q8_K_L if memory serves.

Re: Qwen 3.6 27B is the sweet spot for local development

#678

Post by spider-mario »

mycall wrote: post #613
jasonjmcghee wrote: post #579

Earlier quoted context omitted.

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always…

It is less efficient use of the GPU and uses more electricity overall, no?

Oh no, 0.6 kWh a day!

Re: Qwen 3.6 27B is the sweet spot for local development

#679

Post by this_was_posted »

ThunderSizzle wrote: post #659
androiddrew wrote: post #541

Dual AMD Radeon AI Pro 9700s (600 watts total 64GB of vram) runs Qwen 3.6 27B at FP8 with mtp on vLLM at 50ish TPS for decode. Cards cost $1300 a piece. Enough KV cache to fully max out two concurrent sessions. It was super rough going to get started with them back in January, but right now the cards purrrr and I haven't even tried tuning yet. You need to use a patched vLLM image with aiter but besides that things ar…

Agreed. I have a single 9700 and I'm able to fit Q6 27B at 30tps or Q5 35B at 100tps very easily via llamacpp running vulkan. The results are impressive considering the amount of people trashing AMD and still trying to recommend 3090s. I hope to buy a 2nd one at some point, but I also hate the version hell of vLLM, the R9700, the ROCM version, and Qwen3.6 all not agreeing with each other. I haven't gotten vLLM to run…

I'm getting around 45 tps on a single r9700 for Q6 27B with build b9811 ( using https://github.com/kyuz0/amd-r9700-ai-toolboxes ) with the following parameters:

llama-server -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q6_K -c 135000 -ngl 999 -np 2 -t 16 --temp 0.0 --top-p 0.95 --top-k 20 --min-p 0.00 -b 4096 -ub 4096 --chat-template-kwargs '{"preserve_thinking": true}' -fa 1 --spec-type draft-mtp --spec-draft-n-max 2

Re: Qwen 3.6 27B is the sweet spot for local development

#680

Post by urbsgpw »

UncleOxidant wrote: post #464
girvo wrote: post #393

Earlier quoted context omitted.

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

It would be great if the Gemma folks would release a code-focused model. Probably won't happen, but it's fun to dream.

Ya, doesn't seem to be google's focus at all, right?
Post reply on HN