Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

351–360 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#351

Post by sanderjd »

not_kurt_godel wrote: post #332
sanderjd wrote: post #315

Earlier quoted context omitted.

Continuing to learn new ones, like what? To me, "how do contemporary AI systems work and interact with contemporary hardware and how can I best take advantage of their capabilities?" is the set of skills that are worth learning at this moment. What else is there? New / additional programming languages? New / additional database systems? frameworks? orchestrators? cloud provider / infra tooling? architectural patterns…

Yes, that all tracks, and all of those skills are worth maintaining and improving. Great to tinker with LLMs locally hands-on to learn, and having a powerful enough machine to enable that to a reasonable degree is just one of many reasons why it's worth it. I'm just saying that IMO "how can I best take advantage" lands firmly in the bucket of only cloud-hosted frontier models being worth my time. I would speculate th…

Maybe. I felt this way a year ago and definitely two years ago. But now my sense is that it's played out at this point, and the valuable thing to build expertise on now - precisely because I think it's coming rather than here - is local / open weights / hybrid models and harnesses.

Re: Qwen 3.6 27B is the sweet spot for local development

#352

Post by dofm »

Fr0styMatt88 wrote: post #312
VerifiedReports wrote: post #278

Earlier quoted context omitted.

Exactly. The distinction between the various layers in "AI" systems is pretty vague to the newcomer. What is the "model" vs. the engine "running" it vs. weights? I don't recall any previous tech stack that was barfed onto the scene with so little background or reference material, going from zero to endless undefined jargon... and no primer in sight. For people who demand an understanding of their tools, it's a lot of…

The most unexpected thing for me was kind of philosophical in a ‘holy shit’ way. Cloud models still feel ‘magic’, like you send a request off and get something back, like it’s something ‘special’. I used to joke that ChatGPT might be some kind of mechanical turk underneath. Watching a model run local on your own machine hits different — you realise that yes, it IS just a computer program. Which for me actually makes…

Yep — it hasn't changed how I feel about what LLMs are capable of (and very much not capable of) but this visceral feeling is fascinating.

Like, just watching a computer I already owned act like ChatGPT with the wifi disconnected.

It was the first time I stopped feeling quite so helpless, somehow.

Re: Qwen 3.6 27B is the sweet spot for local development

#353

Post by c7b »

acters wrote: post #212
iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Would the new upcoming AMD AI ryzen halo desktop be a better value offer? or dgx spark? You would have to get a third party reseller/scalper or refurbished mac mini to get 64gb of ram ever since apple stopped selling it.

My 2c: you don't need the Strix Halo desktop, the chip comes in many rigs, most of them cheaper, the performance difference isn't worth it. It used to be half the price of a DGX Spark or a Mac with 128GB RAM. If you can still find it at that price I'd say it's the best bang for your buck. Otherwise, Macs have 2-3x the memory bandwidth of the DGX Spark, depending on the chip, so I'd prefer them. Unless you're planning on building a cluster. The DGX Spark has two 100GB/s connectors, ideal for clustering. But I haven't checked what else you could get for the price of two DGX Sparks.

Re: Qwen 3.6 27B is the sweet spot for local development

#354

Post by nsbk »

chorizo wrote: post #248
oceanplexian wrote: post #207

Earlier quoted context omitted.

If you want to do coding with a local LLM your best bet is a 6 year old Nvidia 3090 which is substantially more powerful than the highest end overhyped Apple product for 1/5th the price.

That’s 24GB VRAM. Not enough to run a 27B model at a useful quant+context size.

I beg to differ. Have a look at this repo with single/double 3090 optimized configs for Qwen and Gema models: https://github.com/noonghunna/club-3090

Re: Qwen 3.6 27B is the sweet spot for local development

#357

Post by overgard »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

I'm running an M5 Max 128GB with Qwen 3.6 and unreal engine in the background and it seems to be ok for me. Quite a power drain if it's not plugged in but I haven't seen any thermal issues.

Re: Qwen 3.6 27B is the sweet spot for local development

#359

Post by Xeoncross »

ctkhn wrote: post #298

I have been running qwen 3.6 35b a3b with opencode on my macbook pro 16" with m3 max and 64gb ram, and it's been great for local planning and coding. To be honest I have been on and off wishing I had future proofed with the 128gb after seeing how powerful 64gb is. On the other hand, I also haven't run up against a wall with a model that is just slightly larger than qwen.

What is the speed on responses? (t/s)

The full 128GB is surely helpful in keeping browsers, editors and other things running since even 20-35GB models + k/v caches can eat up a lot of the core 64GB in my experience.

Re: Qwen 3.6 27B is the sweet spot for local development

#360

Post by SwellJoe »

SkitterKherpi wrote: post #254
chorizo wrote: post #248

Earlier quoted context omitted.

That’s 24GB VRAM. Not enough to run a 27B model at a useful quant+context size.

You can run 8bit 27B models at 24GB, it's definitely enough for the model size.

The 8-bit quantized 27B Qwen 3.6 is 29GB. You absolutely cannot run that entirely on a 24GB GPU.

You could run a 4-bit, which is 16-17GB. But, you'd need a smallish context or you'd need to quantize your KV cache. Something like TurboQuant or RotorQuant might help.

32GB is the lower bound for comfortably running this size model. I'd maybe even say 64GB is right-sized, because a 256k context is nice to have for agentic workflows, and that won't fit on a 32GB card without heavy quantization (but I haven't tried TurboQuant or RotorQuant to know what impact it has on memory use for context).

You could also put some of the model into system RAM, but that defeats the purpose of your argument that a 3090 will outperform a Mac Mini or Mac Studio. If part of a dense model is in system RAM, it absolutely will not outperform a recent unified memory device.

Post reply on HN