Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

631–640 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#631

Post by HSO »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

running potentially sota open-weight models locally only became a thing in fall 2023.

if a hardware cycle takes ~3 years then fall 2026 would be the first possible device generation where apple exploits its advantage with the unified ram architecture.

more realistically, spring 2027, since they probably also needed some time to make up their minds to lean into that on the top end.

that`s also how i would interpret the recent rumors on m6 and m7.

naturally, the cooling and all that will be optimized around that.

so the first devices that are actually intended and designed for this use case will come at the earliest this fall and more likely in q1/q2 next year.

you are basically paying the price now to be on the bleeding (sweating) edge

Re: Qwen 3.6 27B is the sweet spot for local development

#632

Post by wren6991 »

mortenjorck wrote: post #400
aurareturn wrote: post #376

Earlier quoted context omitted.

Don't buy the Mini or Studio. Both have the M4 which lacks the Neural Accelerators, making prompt processing ~3-4x slower.

I assume those don't just work automatically with an off-the-shelf gguf. What do you need in your local inference stack to take advantage of M5's neural accelerators?

Apple muddied the waters by calling them "neural accelerators" but it seems like what they actually added in the M5 generation is tensor instructions for the existing GPU cores. It's not a separate accelerator like the ANE.

llama.cpp's Metal backend does use them when they're available.

Re: Qwen 3.6 27B is the sweet spot for local development

#634

Post by mr_mitm »

Gigachad wrote: post #482
mr_mitm wrote: post #78

Earlier quoted context omitted.

Think commercial. My company invested in a local rig since privacy is important to our customers and sometimes I want to use these models on private data.

Even in that case it would make more sense to put the hardware in a server rack shared with everyone rather than inside macbooks. At any rate it makes a stolen backpack or spilled drink a lot less damaging.

Obviously the rig is not a macbook but indeed a server rack. I'm just saying that we're using this model for local development.

Re: Qwen 3.6 27B is the sweet spot for local development

#635

Post by rvba »

Catloafdev wrote: post #492
bityard wrote: post #396

Earlier quoted context omitted.

Halving the precision of the weights is not a free lunch...

Q8 is virtually lossless. The quantization is much more noticeable around Q4 and below. FP16->Q8 on consumer hardware is 2x the speed at ~99.99% the quality.

Any source that confirms the 99.99% quality?

Re: Qwen 3.6 27B is the sweet spot for local development

#636

Post by adornKey »

cpburns2009 wrote: post #407
SwellJoe wrote: post #360

Earlier quoted context omitted.

The 8-bit quantized 27B Qwen 3.6 is 29GB. You absolutely cannot run that entirely on a 24GB GPU. You could run a 4-bit, which is 16-17GB. But, you'd need a smallish context or you'd need to quantize your KV cache. Something like TurboQuant or RotorQuant might help. 32GB is the lower bound for comfortably running this size model. I'd maybe even say 64GB is right-sized, because a 256k context is nice to have for agenti…

A 32gb card does run it nicely. I use unsloth's UD-Q5_K_XL at 256k context (k/v at q8_0), and get ~67 t/s on a 5090. I still need to look into MTP.

Nice. I used Q4_K_M to have some headroom. But yours seems to fit nicely.

Re: Qwen 3.6 27B is the sweet spot for local development

#637

Post by Dylan16807 »

oceanplexian wrote: post #221
smt88 wrote: post #181

Earlier quoted context omitted.

Economies of scale are a fact of nature and aren’t going to be subverted in the future by even the most advanced local models

Things can get both more expensive and cheaper at scale, hence the term. For example (and relevant to AI) I can generate electricity on my roof at $0.20-25/kWh, batteries included. In California the electric utility can’t offer it cheaper than $0.30-0.50/kWh. Therefore at scale, electricity is actually more expensive. There are many such examples.

Apples and Oranges. The utility uses a weird conflated fee that combines the price of the electricity and the price of connecting your house to the grid. If they split it up your marginal price per kWh would be much less.

Re: Qwen 3.6 27B is the sweet spot for local development

#638

Post by c16 »

jasonjmcghee wrote: post #579
iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always…

Will give this a try later. Enjoy working with A3B Coder, but the heat coming out my 32gb M5 is a lot. This might be the trick - Thanks!

Re: Qwen 3.6 27B is the sweet spot for local development

#639

Post by Tenoke »

amatecha wrote: post #545
Arch-TK wrote: post #493

Earlier quoted context omitted.

It's okay, completely wrong thread for this statement, but I wouldn't voluntarily use current MacOS (no idea if the older variants weren't terrible) over anything but ssh. Worse than Windows 11.

"macOS" (or however they spell it now) is pretty bad, but I'm not sure it's possible Apple could ever possibly produce an OS as bad as Windows 11 lol, it's really surprising to me to see someone suggest it's somehow actually worse?! How many times has an Apple OS wiped your hard drive or otherwise been completely borked from a forced update? I know multiple people personally who have experienced this with Windows 10/…

>How many times has an Apple OS wiped your hard drive or otherwise been completely borked from a forced update

I use Windows and this has never happened to me. I have had Macbooks I cant open to fix/replace something trivial while I can replace any part easily on a Windows PC/laptop though.

Re: Qwen 3.6 27B is the sweet spot for local development

#640

Post by Dylan16807 »

Gigachad wrote: post #473
smt88 wrote: post #181

Earlier quoted context omitted.

Economies of scale are a fact of nature and aren’t going to be subverted in the future by even the most advanced local models

The economies of scale gains are lost because you still have a middle man hosting provider who wants to profit too. Over the long term it's always been better to buy than to rent, even if the renting option is technically more efficient on the GPUs, you don't have to pay some hosting providers profit margin.

If the hosting provider can fit 1000 users onto 100 GPUs, that's enough for quite nice margins and being far cheaper than buying your own GPU.

And for users that aren't running multiple agents 24/7, you should be able to fit a good user:GPU ratio.

Post reply on HN