Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

711–720 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#711

Post by devin »

simplyluke wrote: post #423
devin wrote: post #417

Earlier quoted context omitted.

Implicit in your answer is the belief that they will come back to earth. I wonder how realistic that belief is.

We have decades upon decades of hardware getting dramatically cheaper year over year for the same performance, and ~1 year of the inverse due to dramatic buildout for AI. It's a surprising example of the recency bias to me to assume anything other than the market returning to its historic norm, even if the AI buildout doesn't slow, producers will scale factories to meet that demand.

I look forward to re-evaluating this statement in, what do you say, 12 months from now?

Re: Qwen 3.6 27B is the sweet spot for local development

#712

Post by fancyfredbot »

bigyabai wrote: post #515
astrostl wrote: post #474

Earlier quoted context omitted.

> MacBook Pro M5 128GB RAM 614 GB/s of memory bandwidth > MacMini M4 with 64GB of RAM 273 GB/s of memory bandwidth (also only currently available with 48GB) When it comes to inference speed, you want your model to fit in memory, and then to have as much memory bandwidth as possible. In this case a hypothetical Mini with 1TB of memory would still be over 2x slower with 27-35B models. And FWIW I have an M4 Max MBP 128G…

> When it comes to inference speed, you want your model to fit in memory, and then to have as much memory bandwidth as possible. This is only true when your GPU isn't bottlenecked building a KV cache, which it usually will be on Apple Silicon. The Achilles heel of the M-series chips are their weak, SOC-grade GPU that holds back the Max and Ultra models from having interactive TTFTs on larger models and contexts.

Normally people refer to the compute-bound phase as "prefill". Nothing wrong with saying it's building the kv cache though, it's accurate just unusual.

Re: Qwen 3.6 27B is the sweet spot for local development

#713

Post by edb_123 »

mft_ wrote: post #366
JeremyNT wrote: post #121

Earlier quoted context omitted.

I also don't understand why people in this price bracket are buying Mac laptops instead of desktop computers with GPUs? Just to flex that it's portable?

(I'm not one of the people you're speaking of with a 128gb M5 but) if you want to run one of the medium-sized open-weights models (Qwen 27b, 35b, Gemma 4 26b, 31b) or larger, you get into an interesting optimisation space. * yes, you can run it on an older/smaller GPU plus system RAM but performance will suffer * if you want optimal GPU performance you need the model in VRAM plus context, so 24GB (3090, 4090) or 32GB…

I just recently got into experimenting with local LLMs when I had anyway (for non-LLM reasons) built myself a new desktop system with Intel Ultra 270K-Plus and RTX 5080. With 64GB system RAM and 16GB VRAM. Relatively speaking a high-performing and low-to-moderate cost system.

I wasn't really expecting much from these local open weight models neither when it comes to speed or "intelligence", but my preconceptions were quickly put ashame when I got ollama up and running and pulled my first model. I get a consistent 117-128 t/s with Gemma4:26b-a4b without any tuning (just the default settings), which was much faster than I had expected. Can't wait to dive deeper into this, especially with Qwen3.6 models.

Does anyone's have experience adding a 2nd Nvidia GPU of the same generation but different (slower) model in the same system? Will it give a major boost with larger models, or will the slower card just be a bottleneck? I have an unused RTX 5060 Ti 16GB that I'm considering to install alongside the RTX 5080, but it would necessitate removing some other hardware, so I haven't bothered yet.

Re: Qwen 3.6 27B is the sweet spot for local development

#714

Post by hughw »

bsder wrote: post #437
dofm wrote: post #50

Earlier quoted context omitted.

The maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. I don't know how much serious hands-free agentic coding I will ever do on my MacBook alone, but I do know that I would not have got so far into understanding this without tinkering with local models, llama.cpp, LM Studio, and LM Studio and all th…

> I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. Because it's just huge, exhausting, jargon-drenched, unknowable, and I am over the hill at fifty-plus. Hello, my brother, just know that you have a fellow passenger in life at the same age who thinks the same thing. I agree that the local stuff is helping my understanding a LOT. However, my gut…

You looked into vendors like https://www.runpod.io/?

Re: Qwen 3.6 27B is the sweet spot for local development

#715

Post by b3ing »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

You can use a fan app to ramp up how fast the fans spin instead of the default so you can prevent any throttling

Re: Qwen 3.6 27B is the sweet spot for local development

#717

Post by lambda »

girvo wrote: post #393
SwellJoe wrote: post #342

Earlier quoted context omitted.

I opted to buy a normal 32GB laptop for this very reason. I know how loud and hot the GPUs in my desktop run when running even smallish models like Qwen 27B or Gemma 4 31B (which is a better model for most than Qwen 3.6, despite the benchmarks). I also have a Strix Halo which doesn't get loud, because it has a single huge fan, but it does get hot. So, there's no way a laptop could work as hard as models make them wor…

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision.

I do like Gemma for translation, however.

Re: Qwen 3.6 27B is the sweet spot for local development

#720

Post by jvican »

bblb wrote: post #571
zx76 wrote: post #226

I see a lot of people writing about how expensive the hardware to run these local models is - but see no mentions of the Intel Arc Pro B50/B60/B70 which seem like decent value if you're not interested in Apple kit (as much as anything can be decent value in the current status quo). I just got a B70 with 32GB RAM for the equivalent of $1200 (incl. sales tax and import duties to my non-US location, so presumably it cou…

I got B70 few days ago. Running on CachyOS. 9070XT on PCIe x16 and B70 on the x4. ROCm nightly was pretty easy to setup and get up running. The 9070XT has been a decent card for my use cases. But the SYCL ecosystem versions. Absolutely horrendous and everything is hundred commits behind. Vulkan is probably the only way forward with this card.

This is very interesting. I've only heard bad things about B70's performance with Vulkan, for example this Reddit link mentioning a Phoronix review: https://www.reddit.com/r/LocalLLaMA/comments/1sgdt7t/comment...
Post reply on HN