Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

31–40 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#31

Post by dofm »

rhgraysonii wrote: post #17
dofm wrote: post #14

Earlier quoted context omitted.

You might be interested in Ornith 1.0 9B, which is a new intriguing post-training of Qwen 3.5 9B. Qwen 3.6 27B will run in full offload with a 4-bit quantisation in 64GB on an M1 Max. It is quite slow. I don't know about 48GB but 64GB should be enough.

Thanks! I was thinking of doing the 128gb to have some future proofing. I figure at this point, it's akin to a mechanic keeping great tools around, when it comes to having this sort of homelab and exposing it for your own uses. And great practice for building the next era of user facing computing that will be around as this proliferates.

I would not buy a 64GB model again, probably, if this were to remain particularly important to me. But I gather memory bandwidth is pretty important here.

So for example I'd favour a used M1 Max over a used M2 Pro, at least based on my naïve understanding. Not quite sure where the balance changes.

There appear to be some hardware improvements with the M3 and up regarding the Apple Neural Engine which I'd hope would show up in MLX performance; I remember seeing some optimisations in image generation models that are only possible on later hardware.

The GPU cores are progressively better I believe, but the memory bandwidth is lower. Though perhaps the M4 can get closer to actually saturating said bandwidth.

(And I must reiterate that my understanding of this stuff is pretty naïve.)

Re: Qwen 3.6 27B is the sweet spot for local development

#32

Post by Catloafdev »

RedCinnabar wrote: post #25

Call me back when you can run these models on 16GB of RAM and any recent i5/i7. Until then, there’s no point on using these toy models.

Hello, it's the internet calling, today is that day.

https://github.com/ikawrakow/ik_llama.cpp

Edit: it's gonna be slow if you're not using any VRAM. But it's possible. Software isn't going to speed that up anytime soon, it's just a hardware bandwidth limit.

Re: Qwen 3.6 27B is the sweet spot for local development

#33

Post by someperson »

Insanity wrote: post #30
bensyverson wrote: post #26

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs. [0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

But you have to factor in that this device will last you 5-10 years. That said, I wouldn't spend almost $7k USD on this macbook lol.

In 5-10 years, incremental cloud tokens will be far cheaper (likely but not guaranteed).

Re: Qwen 3.6 27B is the sweet spot for local development

#34

Post by kllrnohj »

0x0000000 wrote: post #19

> ... on my Macbook Max M5 128 GB Local development for who ? How many of y'all are rocking 128GB of memory? Am I reading Apple's site correctly that it's a $10,000 laptop?

You don't need nearly that much RAM to run Qwen 3.6 27B, though. qwen3.6:27b-q4_K_M is only 17GB, for example.

Re: Qwen 3.6 27B is the sweet spot for local development

#36

Post by petilon »

Insanity wrote: post #30
bensyverson wrote: post #26

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs. [0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

But you have to factor in that this device will last you 5-10 years. That said, I wouldn't spend almost $7k USD on this macbook lol.

Memory requirements of newer models will increase, so while the hardware may last 10 years it won't be able to run the latest models for 10 years.

Re: Qwen 3.6 27B is the sweet spot for local development

#37

Post by h4ny »

bensyverson wrote: post #26

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs. [0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

[flagged]

Re: Qwen 3.6 27B is the sweet spot for local development

#38

Post by sleepyeldrazi »

dofm wrote: post #8
HotGarbage wrote: post #4

And AI companies will continue to buy up all the silicon to make this prohibitively expensive to run at home.

It will run (somewhat slowly) on a five year old M1 Max with 64GB RAM. Personally I prefer the 35B MoE model, which is fast enough to be interactively useful, and capable, but I would probably use the 27B if I wanted to generate whole applications like that. I am unconvinced that most "local" AI applications need anything much more powerful than the Gemma 4 12B model. Local agentic coding is a small niche, but there…

I need to ask, since I have desperately wanted to make Gemma 4 12B work, but im not sure if its the quant (i usually up it to q8, which is a lot higher than iq4_nl that i use for 3.6 27B) or the model itself, but it just starts confusing itself really quickly when I give it coding tasks. And quickly starts failing tool calls.

I really want to have a model that i can run locally on my 24gb m4 pro mbp for when i don't have internet to connect to my 3090 running the qwen, and i love how gemma 4 models 'feel', but i can't make them be competent. I am in the middle of finetuning both qwen3.5 9B and gemma 4 12B just to try and make those bridge closer to 27B for coding/agentic tasks (and am trying to ternarize and DQT 27B so that it fits in ~9gb pre-KV).

How do you run the gemma? What do you use it for (and in what harness), maybe llama.cpp and pi-mono just aren't for this model and that's what i'm doing wrong.

Re: Qwen 3.6 27B is the sweet spot for local development

#40

Post by dannyw »

blobbers wrote: post #13

How does llama.cpp use the GPU efficiently as opposed to MLX? Is there any way to use MLX and GPU at the same time? Or does memory become a big problem? TBH, I never understood Apple hyping these neural cores because I didn't think anyone actually uses them except maybe certain photo/video editing software. If I can generate voice at the same time as video, that would be useful.

Llama.cpp uses the GPU very effectively because inference of LLMs is very rudimentary and basically as simple as your GPU memory bandwidth. That's essentially the baseline performance ceiling, with model-specific optimisations like MTP potentially increasing it.

The neural cores aren't suitable for LLMs/transformers and isn't used in LLM inference. On the M5 and later chips, it comes with neural accelerators, aka Tensor Cores, which speed up the 'prefill' (i.e. processing your context window) part, but don't do anything for inference.

The MLX vs GGUF debate is mostly irrelevant. The GGUF pathways are optimised for apple silicon to the extent of practically identical performance to MLX. MLX is just one way of using Apple GPUs, it comes with many optimisations in the box, but they're not hard and they're no longer MLX-exclusive.

Post reply on HN