Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

641–650 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#641

Post by pantulis »

somewhatrandom9 wrote: post #439
iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Try using DwarfStar 4 and use the --power flag: https://github.com/antirez/ds4#reducing-heat-power-usage-and...

DwarfStar is the only thing I've run that doesn't try and make my Mac Studio 128GB take off. Yes, it gets hot while doing inference but quickly cools down when idling, something I haven't experienced with Ollama, LMStudio or OMLX.

Re: Qwen 3.6 27B is the sweet spot for local development

#643

Post by Gigachad »

Dylan16807 wrote: post #640
Gigachad wrote: post #473

Earlier quoted context omitted.

The economies of scale gains are lost because you still have a middle man hosting provider who wants to profit too. Over the long term it's always been better to buy than to rent, even if the renting option is technically more efficient on the GPUs, you don't have to pay some hosting providers profit margin.

If the hosting provider can fit 1000 users onto 100 GPUs, that's enough for quite nice margins and being far cheaper than buying your own GPU. And for users that aren't running multiple agents 24/7, you should be able to fit a good user:GPU ratio.

Maybe. The economics work out better than for game streaming. When I looked in to game streaming it ended up being cheaper to buy over the long term. Though games tend to use 100% of the hardware for hours, and they tend to all be used at the same hours of the day and have to be hyper local for latency reasons. Something LLMs don’t have issues with.

Re: Qwen 3.6 27B is the sweet spot for local development

#644

Post by kamranjon »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

I completely disagree, it is probably the best platform currently for this - and the way I run it is as a server with tailscale accessible from my coding machine (same as you suggest here) - the difference is that you can stop the server, use it as a video editing rig on a whim, or use it for training instead of inference (yes PyTorch has caught up and Metal is a great platform for this now).

It’s just so flexible, and I even use it in agent mode (ds4) directly on the machine as well sometimes (it’s really not that bad, I’m often running inference for small side projects on my couch), if there is another machine that can do all of this and still function as one of the more ergonomic, well built, and compact laptops out there, I’d love to hear what it is cause I’d likely be interested!

Re: Qwen 3.6 27B is the sweet spot for local development

#645

Post by pimeys »

bensyverson wrote: post #26

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs. [0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

Yes. It is very expensive now. I'm still so so happy I decided last summer to bite the bullet and pre-ordered the Framework Desktop 128GB model.

I paid 2424 euros in total for this machine. And it can easily run the models discussed in the comments and in the article. It's tiny, and runs CachyOS like a champ. Over 4000 euros less than the price you listed.

We can all send a thank you letter for our friendly billionaires such as Sam Altman for the price situation we're in today: https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram...

Re: Qwen 3.6 27B is the sweet spot for local development

#646

Post by ljosifov »

kroaton wrote: post #478
ljosifov wrote: post #255

Running 27B dense model on M5 128GB is ok, but one can do better. On M5 128GB one can make use of the ram and use sparse MoE. For example, DeepSeek-V4-Flash will fit, served by DwarfStar ( https://github.com/antirez/ds4 ). One will probably improve 2x the token/sec speed, given DS4F 13B activated params in the MoE are ~1/2 of the ~27B of the dense Qwen. 27B Of the Qwen fit even on a cheaper 24GB card, e.g. amd 7900xt…

"DeepSeek-V4-Flash will fit" At Q2, 2bit? Lobotomized to death.

Hobbled - but not to death, the few times I use it (usually on a plane). I tried 2bit of a 20% REAP reduced experts. :-O That's the biggest that fits on my own h/w (3yrs old M2 Max 96gb). It's coherent, it does work, doesn't fall apart on casual use. IDK if better than dense 27b. Think 27b was slower on the same h/w. DS4F has got 1M context window. Nowadays with weeks long run hermes sessions, I get to 300k-400k context depths easily. The speed decline profile of DS4F with context depth increase is superior to any other model I try. (I try them all - love this stuff) Only previous model coming close on that is nemotron-cascade-2 (only 30b-a3b) - that also has 1M context window.

Re: Qwen 3.6 27B is the sweet spot for local development

#647

Post by hypfer »

dofm wrote: post #404
QuercusMax wrote: post #343

Earlier quoted context omitted.

Yeah, it's been fun for me running models (mostly Qwen 3.6 27B) on my 48GB M4 MacBook Pro. When i'm using it to run models, it's basically unusable for anything else - I actually do the work on my Macbook Neo. Took me a while to figure out why the models couldn't figure out how to make tool calls - because LMStudio by default uses a 32K input window, which is smaller than OpenCode's prompt, so half of the instruction…

Yes — there is a setting for that isn't there. And as soon as you realise there's a setting for that, you have new knowledge. Qwen barely needs any of Opencode's prompt, in my experience; I think I cut it down to about three general lines I found by googling. Mainly you need only a pre-amble to make sure that the plan mode, plan switch and build mode prompt fragments make sense. Gemma 4 also needs almost nothing at a…

What are those 3 lines you've cut it down to?

Re: Qwen 3.6 27B is the sweet spot for local development

#648

Post by meta-level »

why does everyone imply you need a $10k laptop which then starts burning when you run Qwen 3.6? Get any other system with enough VRAM for a third of the price. Framework Desktop (Strix Halo 128GB) still costs under 4k nowadays, is nearly silent even on 100% GPU + CPU. (also it gets only slightly 'warm', but with a desktop you don't care anyway, I guess).

Re: Qwen 3.6 27B is the sweet spot for local development

#650

Post by Aperocky »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Thank you - I was very close but thanks to chores and availability haven't pulled the trigger. You are very convincing.
Post reply on HN