Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

731–740 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#731

Post by the_lucifer »

html5cat wrote: post #618
jasonjmcghee wrote: post #579

Earlier quoted context omitted.

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always…

Awesome idea! Will try it out. Wish there was a way to enable low power on a per-app basis. Scrolling and reading on low power mode is really annoying.

> Wish there was a way to enable low power on a per-app basis.

Since you can control the low power mode setting from the command line: `sudo pmset -a lowpowermode 1`.

It should be pretty straightforward to hook this up to Hammerspoon[1] using hs.application.frontmostApplication() to apply the setting based on whatever foreground application you choose.

Thinking out loud, that being said, the necessity of sudo might make this slightly more complex. An always on background admin agent might be needed I suppose to bypass the password prompts (or add pmset to the sudoers file, if you prefer).

[1]: https://www.hammerspoon.org/

Re: Qwen 3.6 27B is the sweet spot for local development

#732

Post by skolos »

edb_123 wrote: post #713
mft_ wrote: post #366

Earlier quoted context omitted.

(I'm not one of the people you're speaking of with a 128gb M5 but) if you want to run one of the medium-sized open-weights models (Qwen 27b, 35b, Gemma 4 26b, 31b) or larger, you get into an interesting optimisation space. * yes, you can run it on an older/smaller GPU plus system RAM but performance will suffer * if you want optimal GPU performance you need the model in VRAM plus context, so 24GB (3090, 4090) or 32GB…

I just recently got into experimenting with local LLMs when I had anyway (for non-LLM reasons) built myself a new desktop system with Intel Ultra 270K-Plus and RTX 5080. With 64GB system RAM and 16GB VRAM. Relatively speaking a high-performing and low-to-moderate cost system. I wasn't really expecting much from these local open weight models neither when it comes to speed or "intelligence", but my preconceptions were…

I'd say adding another 16Gb gpu would be worth it - you'd be able to run larger model/larger context all within gpu's. It would give you more options of what you can run fast. Your current model probably doesn't run completely from GPU (depending on quants I don't think you can squeeze Gemma4:26b into 16Gb vram), so you already have some layers running on gpu and some on cpu. If you add another gpu you might be able to move all layers to vram which should speed up things for you. The layers calculations happen on whatever gpu's it sits, so the layers that are already on your rtx5080 would compute same, but the layers that currently your cpu handles will be computed with faster vram/compute of rtx5060.

Re: Qwen 3.6 27B is the sweet spot for local development

#733

Post by razster »

bensyverson wrote: post #26

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs. [0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

I'm running it on my 4070 12gb with 96gb mem, I'm very happy with the results even if I have to wait a couple minutes for results. To me this is far better than I expected and will continue to use it and improve with skills.md. Pi.dev is amazing by the way.

Re: Qwen 3.6 27B is the sweet spot for local development

#734

Post by cpburns2009 »

etdznots wrote: post #723
cpburns2009 wrote: post #465

Earlier quoted context omitted.

Looping is a common problem with the Qwen models. I've had good luck using --repeat-penalty=1.1 with llama.cpp and 27B. vLLM should have a similar option.

This is the default value!

Llama.cpp defaults to 1.0 (disabled) and so does vLLM. It looks like only ollama defaults to 1.1.

Re: Qwen 3.6 27B is the sweet spot for local development

#736

Post by [deleted] »

hypfer wrote: post #647
dofm wrote: post #404

Earlier quoted context omitted.

Yes — there is a setting for that isn't there. And as soon as you realise there's a setting for that, you have new knowledge. Qwen barely needs any of Opencode's prompt, in my experience; I think I cut it down to about three general lines I found by googling. Mainly you need only a pre-amble to make sure that the plan mode, plan switch and build mode prompt fragments make sense. Gemma 4 also needs almost nothing at a…

What are those 3 lines you've cut it down to?

[deleted]

Re: Qwen 3.6 27B is the sweet spot for local development

#737

Post by 1-6 »

zx76 wrote: post #226

I see a lot of people writing about how expensive the hardware to run these local models is - but see no mentions of the Intel Arc Pro B50/B60/B70 which seem like decent value if you're not interested in Apple kit (as much as anything can be decent value in the current status quo). I just got a B70 with 32GB RAM for the equivalent of $1200 (incl. sales tax and import duties to my non-US location, so presumably it cou…

A 3090 never came with 32gb of vram

Re: Qwen 3.6 27B is the sweet spot for local development

#738

Post by kmacdough »

the_lucifer wrote: post #731
html5cat wrote: post #618

Earlier quoted context omitted.

Awesome idea! Will try it out. Wish there was a way to enable low power on a per-app basis. Scrolling and reading on low power mode is really annoying.

> Wish there was a way to enable low power on a per-app basis. Since you can control the low power mode setting from the command line: `sudo pmset -a lowpowermode 1`. It should be pretty straightforward to hook this up to Hammerspoon[1] using hs.application.frontmostApplication() to apply the setting based on whatever foreground application you choose. Thinking out loud, that being said, the necessity of sudo might m…

Unfortunately doesn't cover scrolling HN while the agent toils away.

Re: Qwen 3.6 27B is the sweet spot for local development

#739

Post by stiray »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

I am using MacBook Pro M4 with 64GB of RAM and I have it on direct path of air conditioning airflow, 40ish cm from the device, while running LM Studio opened to network. No noise, not hot to the touch.

Using linux for actual work on my workstation.

Re: Qwen 3.6 27B is the sweet spot for local development

#740

Post by bigyabai »

mycall wrote: post #613
jasonjmcghee wrote: post #579

Earlier quoted context omitted.

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always…

It is less efficient use of the GPU and uses more electricity overall, no?

Yes, this is a tradeoff that foregoes the efficiency of race-to-idle.
Post reply on HN