Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

501–510 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#501

Post by prima-facie »

roadside_picnic wrote: post #365
prima-facie wrote: post #173

Earlier quoted context omitted.

> The biggest thing to watch out for is not just RAM/VRAM but memory bandwidth. You can try to "future proof" yourself with lots of RAM, but if it's 400 GB/S you're still constrained to smaller models. I'm thinking of getting a SoC machine with 128GB RAM but the bandwidth is limited to 256 GBps. Would you even consider such a machine a decent investment, or should I wait for the newer gen of chips? Thanks!

It depends on your use case. There's a lot of hype around machines like the DGX spark (I'm assuming this is the type of device you're referring to) because they look awesome, and are priced reasonably well. However all of these have notoriously low memory bandwidth despite the high ram. These devices, especially the DGX line, are fantastic if you are interested in low-level CUDA programming. The DGX spark can be used…

Thanks for the detailed response, I really appreciate it.

What I had in mind was an AMD Strix Halo machine, but it seems to have none of the advantages you mentioned. It's neither high bandwidth, nor does it have CUDA support, nor does it have support from the big OEMs. All the boards are from relatively obscure Chinese vendors.

It seems like all the major OEMs have rallied behind Nvidia, if you look at the upcoming RTX Spark laptops.

Re: Qwen 3.6 27B is the sweet spot for local development

#502

Post by bitexploder »

coder543 wrote: post #471
bitexploder wrote: post #129

Earlier quoted context omitted.

For a MBP I have 48 GB of RAM M5 Pro. It runs at about 12-14 t/s at Q4, you could probably optimize it further. RAM is not a limitation but overall memory bandwidth. Q8 is slower. 35B A3B Qwen is quite speedy, but a little less accurate. With Qwen 3.6 27B dense I can squeeze a 9B parameter model and use that for fast analysis or code scanning while 27B is churning on a task in the background. It is tight, but totally…

> For a MBP I have 48 GB of RAM M5 Pro. It runs at about 12-14 t/s at Q4 Are you running with MTP enabled? I have seen some people on M5 hardware report 20+ t/s on Qwen3.6-27B using MTP... and I think that was a regular M5, not even M5 Pro.

Nope. MLX in LMStudio. The simplest config with zero tuning effort.

Re: Qwen 3.6 27B is the sweet spot for local development

#503

Post by andrekandre »

jeffybefffy519 wrote: post #479
bensyverson wrote: post #26

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs. [0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

I still dont trust the Anthopic and OpenAI are not training on my code. I even just thinking keeping track of what code you have received in prompts and to train/not train on it seems like an impossibly difficult task.

am i right in assuming your code is closed-source?

i'd expect anything on github for example to be already in their training set or is training on actual usage more useful to them?

Re: Qwen 3.6 27B is the sweet spot for local development

#504

Post by jtbaker »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Nope, have both these machines, can confirm the M5 max blows the M4 mini away. It does get hot, but I use it mostly with an external monitor and keyboard. Conceptually I like the headless model better with a workstation, but work was buying the M5 and can't get it in any other form factor at the monute.

Re: Qwen 3.6 27B is the sweet spot for local development

#505

Post by ekianjo »

girvo wrote: post #393
SwellJoe wrote: post #342

Earlier quoted context omitted.

I opted to buy a normal 32GB laptop for this very reason. I know how loud and hot the GPUs in my desktop run when running even smallish models like Qwen 27B or Gemma 4 31B (which is a better model for most than Qwen 3.6, despite the benchmarks). I also have a Strix Halo which doesn't get loud, because it has a single huge fan, but it does get hot. So, there's no way a laptop could work as hard as models make them wor…

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

gemma is also worse for tool calling. not just coding

Re: Qwen 3.6 27B is the sweet spot for local development

#506

Post by gnerd00 »

lee_ars wrote: post #323
acters wrote: post #212

Earlier quoted context omitted.

Would the new upcoming AMD AI ryzen halo desktop be a better value offer? or dgx spark? You would have to get a third party reseller/scalper or refurbished mac mini to get 64gb of ram ever since apple stopped selling it.

I'm currently fiddling with a DGX Spark and Qwen3.6-35B-A3B (specifically Qwen3.6-35B-A3B-NVFP4 under vLLM, with EAGLE3 speculative decoding via eagle3-dogacel-vllm), and it's pretty okay in terms of smarts. The speed is relatively usable at about 50 tok/sec with a 256k context window, and it's definitely smart enough to one-shot some basic coding tasks. I had it doing reverse engineering/disassembly of some ancient…

`llama-server` looping mitigations --repeat-penalty something greater than 1.0, set reasoning/thinking OFF explicitly, prefer a gguf with more than 4bit quant

Re: Qwen 3.6 27B is the sweet spot for local development

#507

Post by coder543 »

bitexploder wrote: post #502
coder543 wrote: post #471

Earlier quoted context omitted.

> For a MBP I have 48 GB of RAM M5 Pro. It runs at about 12-14 t/s at Q4 Are you running with MTP enabled? I have seen some people on M5 hardware report 20+ t/s on Qwen3.6-27B using MTP... and I think that was a regular M5, not even M5 Pro.

Nope. MLX in LMStudio. The simplest config with zero tuning effort.

Unsloth Studio is also very low effort, and a lot better than LM Studio in my opinion. (Performance, compatibility with Gemma 4, actually open source, etc.)

Re: Qwen 3.6 27B is the sweet spot for local development

#508

Post by Havoc »

drnick1 wrote: post #466

Has anyone managed to cleanly integrate Web search into local models (run with llama.cpp)? The biggest limitation of the class of models that fit into one or two consumer GPUs is that they lack world knowledge, but presumably this can be remedied by enabling access to use the Internet.

Searxng is the ghetto solution. Commercial uruky is good. Basically Kagi except you can also run api calls over it

Neither is going to return much knowledge. Basically just relevant url so you need a second tool to grab them and there bot walls get tricky

Re: Qwen 3.6 27B is the sweet spot for local development

#509

Post by spaqin »

Otternonsenz wrote: post #91

Is there any hope for people that cant even run 27B parameters, Qwen3.6 or otherwise? Are there any quantized models that do well with tool calling at smaller parameter sizes? I do not have a crazy rig, a modest gaming one at that, but in trying to understand more about agents and their capabilities, I am SOL with my 16 GB of RAM and 8GB of VRAM. I can get most small, non tool calling models to perform well, but I've…

I got a 32GB of RAM and a 6GB VRAM card; tried both 27B and 35B, with pi. And it's a laptop. Speed isn't exactly a concern for me, I can enjoy the real life while the agent is doing its thing. And while they appear smart enough on the first glance, once it reads a file that's more than 100 lines it loses all memory of anything I asked it to do. The lack of failure state or any indication what might be wrong here is just frustrating. Guess local models aren't for me, unless I move to Silicon Valley and redeem my free MacBook at a local Startbucks.

Re: Qwen 3.6 27B is the sweet spot for local development

#510

Post by porphyra »

lee_ars wrote: post #403
porphyra wrote: post #67

Earlier quoted context omitted.

You can also run Qwen 3.6 27B dense model on DGX Spark with comparable performance [1][2] for about $4000 (Asus Ascent GX10 is $3999 at various retailers). In theory you can also get 48GB of VRAM with, say, two 3090s, but it will take up a lot of space and generate a lot of heat compared to the Macbook Pro and GB10. [1] https://x.com/MiaAI_lab/status/2070859135399182444 [2] https://github.com/MiaAI-Lab/Qwen3.6-27B-NV…

The tweet you link shows "Qwen 3.6 35b NVFP4 - 256k ctx, 110 tok/s", but I'm getting only half that, around 50 tok/sec, on a DGX Spark with Qwen3.6-35B-A3B-NVFP4 (via vLLM) plus speculative decode w/EAGLE3. I'd be ecstatic to see 110 tok/sec and I wish they had some more sourcing for the exact config, because it's double what I'm getting. edit - after actually reading the tweets (had to use xcancel) and visiting the…

I think Atlas might also be slightly faster than vLLM:

https://flowtivity.ai/blog/120-tok-s-1m-context-private-ai-d...

Post reply on HN