Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

481–490 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#481

Post by Gigachad »

btbuildem wrote: post #389
doodlesdev wrote: post #83

I feel like I'm going insane seeing people buy these 128gb MBP for thousands of dollars to run models that are objectively much worse than SOTA and spending so much more. The amount spent on a 128gb M5 MAX can buy you a damned new car here. What the hell am I missing? Are developers in other countries living in such different worlds? (I'm aware the price is, in absolute terms, more expensive where I live compared to…

I think it's silly to go for a laptop form factor. Last fall I put together a workstation with two second-hand 3090s in it (paid $850CDN each, now the best I can find is $1200). With 48GB VRAM it's reasonable - and I've been using Qwen 3.6 27B for various tasks around building KGs from text corpora / reasoning about them. I've ran comparisons against everything that's available on OpenRouter (well, as of few weeks ag…

I think it still makes sense to wait. Hardware is currently hyper expensive and cloud models are subsidized. Waiting 2 years or so once memory prices have dropped and datacenters start wanting a profit would get you a usable setup that's more economical.

Re: Qwen 3.6 27B is the sweet spot for local development

#482

Post by Gigachad »

mr_mitm wrote: post #78
0x0000000 wrote: post #19

> ... on my Macbook Max M5 128 GB Local development for who ? How many of y'all are rocking 128GB of memory? Am I reading Apple's site correctly that it's a $10,000 laptop?

Think commercial. My company invested in a local rig since privacy is important to our customers and sometimes I want to use these models on private data.

Even in that case it would make more sense to put the hardware in a server rack shared with everyone rather than inside macbooks.

At any rate it makes a stolen backpack or spilled drink a lot less damaging.

Re: Qwen 3.6 27B is the sweet spot for local development

#483

Post by Gigachad »

kgeist wrote: post #409
guax wrote: post #241

Earlier quoted context omitted.

Its so funny, these "toy models" would be the wet dreams of researchers not 5 years ago. Progress marches without mercy.

Yeah people don't realize these "toy models" now completely destroy gpt-4o on most tasks, and no one called gpt-4o a toy model back in the day... It was OpenAI's flagship model from 2024 to 2025.

Tbh in 2024 most were calling these models useless for programming and a scam. It wasn't until this year things really changed. My experience with Qwen 3.6 is it can do things, and it's super impressive it can do things, but it's not any more productive than doing it myself.

Re: Qwen 3.6 27B is the sweet spot for local development

#484

Post by c-hendricks »

c-hendricks wrote: post #143
dofm wrote: post #116

Earlier quoted context omitted.

> - opencode with it's webui Have you tried Paseo? I have opencode in a VM, and the paseo daemon running in the VM, and then the Paseo Mac app. Really nice. (You can also use the Opencode GUI to frame a remote opencode web interface)

You can also just add OpenCode web as a PWA, if that's what you mean by "frame". I'm gonna check out paseo, but am not looking forward to all the ram the agent needs + all the ram paseo needs

Have checked out Paseo, not sure what it offers over opencode web though. Definitely seems great if you're using other harnesses, but it seems like all it has over opencode web is split views and native apps. Neither of those really matter to me, plus you lose some opencode goodies. The preview urls are a neat idea, but our dev servers at work are mostly port independent and required to be on a certain subdomain for auth.

Re: Qwen 3.6 27B is the sweet spot for local development

#485

Post by SwellJoe »

UncleOxidant wrote: post #464
girvo wrote: post #393

Earlier quoted context omitted.

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

It would be great if the Gemma folks would release a code-focused model. Probably won't happen, but it's fun to dream.

The Ornith folks say they're doing that, but haven't released the Gemma-based 31b yet (https://github.com/deepreinforce-ai/Ornith-1). But, also, the Qwen-based 35b MoE Ornith version performs worse than Qwen 3.6 and Qwen AgentWorld on my benchmarks (which are focused on finding security bugs, so not exactly the same as agentic coding, but closely related skills).

That said, the reason they're able to release Ornith branded post-trains of both Gemma and Qwen is because they're open weights under a friendly license. Someone, not just Google, could make a coding focused Gemma post-train. I don't think it's actually much weaker than Qwen 3.6 for coding; Gemma 4 31b outperforms Qwen 3.6 27b by a wide margin on security bug hunting (at least for the specific bugs in my benchmarks, which are mostly relatively difficult bugs from the Mythos-reported bugs).

I'd really love to see a bigger MoE from Google, though. A 70b or 120b MoE would likely be super fun.

Re: Qwen 3.6 27B is the sweet spot for local development

#486

Post by nok22kon »

zingar wrote: post #192
nok22kon wrote: post #120

Earlier quoted context omitted.

you can probably run Gemma4 26B on your card also at 4 bit. World of a difference compared with 12B.

Where does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error?

paper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bit

https://arxiv.org/abs/2212.09720

Re: Qwen 3.6 27B is the sweet spot for local development

#487

Post by coder543 »

lee_ars wrote: post #323
acters wrote: post #212

Earlier quoted context omitted.

Would the new upcoming AMD AI ryzen halo desktop be a better value offer? or dgx spark? You would have to get a third party reseller/scalper or refurbished mac mini to get 64gb of ram ever since apple stopped selling it.

I'm currently fiddling with a DGX Spark and Qwen3.6-35B-A3B (specifically Qwen3.6-35B-A3B-NVFP4 under vLLM, with EAGLE3 speculative decoding via eagle3-dogacel-vllm), and it's pretty okay in terms of smarts. The speed is relatively usable at about 50 tok/sec with a 256k context window, and it's definitely smart enough to one-shot some basic coding tasks. I had it doing reverse engineering/disassembly of some ancient…

Compared to a dynamic quant like Unsloth's UD-Q4_K_XL, which keeps some important parameters in higher precision, a basic NVFP4 quant seems to do a lot more damage to the model unless it is carefully calibrated.

I would recommend using llama-server if you're just on a single Spark. You get access to dynamic quants like that more easily, the performance is not that different from vLLM most of the time these days, and it is much faster and easier to switch between models.

As far as intelligence goes, Qwen3.6-27B is much smarter than the 35B-A3B model, but that's also not the sort of thing to argue with an AI model about in the first place. Just open a new chat and try again.

Gemma-4-31B is not as good at agentic use cases as Qwen3.6-27B, but it is a fairly balanced model overall, and worth trying out too. Its MTP can nearly triple the performance of the model, where the benefits of MTP or Eagle seem more limited for Qwen3.6-27B in my testing, maybe doubling the speed.

Re: Qwen 3.6 27B is the sweet spot for local development

#490

Post by mwowow »

drnick1 wrote: post #466

Has anyone managed to cleanly integrate Web search into local models (run with llama.cpp)? The biggest limitation of the class of models that fit into one or two consumer GPUs is that they lack world knowledge, but presumably this can be remedied by enabling access to use the Internet.

Working fine with LM Studio + Web search plugin
Post reply on HN