Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

761–770 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#761

Post by edb_123 »

skolos wrote: post #732
edb_123 wrote: post #713

Earlier quoted context omitted.

I just recently got into experimenting with local LLMs when I had anyway (for non-LLM reasons) built myself a new desktop system with Intel Ultra 270K-Plus and RTX 5080. With 64GB system RAM and 16GB VRAM. Relatively speaking a high-performing and low-to-moderate cost system. I wasn't really expecting much from these local open weight models neither when it comes to speed or "intelligence", but my preconceptions were…

I'd say adding another 16Gb gpu would be worth it - you'd be able to run larger model/larger context all within gpu's. It would give you more options of what you can run fast. Your current model probably doesn't run completely from GPU (depending on quants I don't think you can squeeze Gemma4:26b into 16Gb vram), so you already have some layers running on gpu and some on cpu. If you add another gpu you might be able…

Thanks! I'm seeing a 10/90 split between CPU/GPU with gemma4:26b, so I guess there's at least something to win there by adding the other GPU. And perhaps something to win by connecting the monitor to the iGPU instead to free up VRAM, from what I gather.

Just in case someone should be interested in how a consumer PC setup like this performs, still using only 1x RTX 5080 + 64GB system RAM and Intel Ultra 270K-Plus; I tested Qwen3.6:35b-a3b now (using ollama and default settings) and I'm getting around ~86 t/s. The lowest I've seen so far is 70 t/s. The CPU/GPU split with 35b is 39/61% (with 4K 165 fps monitor connected to 5080, so there's probably some room for optimization here by moving it to the iGPU).

Best thing is that this setup is basically dead silent (it could, hypothetically speaking, be running in my bedroom just fine, and I'm a light sleeper).

Re: Qwen 3.6 27B is the sweet spot for local development

#762

Post by aand16 »

layer8 wrote: post #42
aand16 wrote: post #3

I've come from the future to say Qwen 3.7 27B is just around the corner and slaps!

Are RAM prices down?

Never did. It was a while back, but in the great Chips War the Micron and Hynix fabs got nuked to atoms. So all memory was confiscated for the war effort.

But now that things are finally turning normal, everybody gets a daily token allowance for GeminiGPT 7.2 and you can even boost your ratio if you volunteer your bioenergy to the local GPT microdatacenter (cycling or run on the H-wheel for an hour gets x1.02 output tokens for the day).

Re: Qwen 3.6 27B is the sweet spot for local development

#763

Post by simplyluke »

devin wrote: post #711
simplyluke wrote: post #423

Earlier quoted context omitted.

We have decades upon decades of hardware getting dramatically cheaper year over year for the same performance, and ~1 year of the inverse due to dramatic buildout for AI. It's a surprising example of the recency bias to me to assume anything other than the market returning to its historic norm, even if the AI buildout doesn't slow, producers will scale factories to meet that demand.

I look forward to re-evaluating this statement in, what do you say, 12 months from now?

I’ll toss $10k in the s&p and you buy the rig and we’ll see who feels like they made a better call?

Re: Qwen 3.6 27B is the sweet spot for local development

#764

Post by alemanek »

v1ne wrote: post #623
alemanek wrote: post #564

Earlier quoted context omitted.

If your workflow benefits from the speed it quickly pays for itself when factoring in developer salaries here in the US. I recently switched companies and they bought me an M5 Max 128GB as my dev machine. Builds and local test runs are 3 times faster than the Windows laptop option. The machine will pay for itself just based on that within 3 months. I can spin up a local kubernetes cluster and do full integration test…

Don't worry. Once IT Security discovers that they miss their trusty endpoint security products on your Mac, they'll add it and you'll be in the same ballpark as the Windows machine. Been there, received that, and learnt that Microsoft Defender exists for macOS, too.

They have all the endpoint protection stuff on this machine. It just seems to run better. Or it could just be that the M5 is doing laps around the Windows workstations but I didn’t think they were that far off looking at the spec sheets. They didn’t cheap out on the Windows side.

Either way enjoying this machine.

Re: Qwen 3.6 27B is the sweet spot for local development

#765

Post by victorbjorklund »

827a wrote: post #518
iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

Apple does not sell a 64GB variant of the M4 Mac Mini. IIRC they never have; its always capped out at 48GB. If you were planning on getting an M5 128GB; just get a DGX Spark (~$4500) or a 5090-equipped machine (~$4500) plus a Macbook Air (~$1500). You'll come in below the M5 Max 128 pricing (~$6700+ USD) and be happier for it.

My MacMini M4 pro has 64gb…

Re: Qwen 3.6 27B is the sweet spot for local development

#766

Post by beacon294 »

CMay wrote: post #751
beacon294 wrote: post #748

Earlier quoted context omitted.

I find that fp8 cache can be pretty bad in vllm but works fine in llama.cpp. I don't know why, but I plan to review the implementations.

Llama.cpp implemented some rotation optimizations for quantized kv cache to improve the preservation of attention quality or similar, after everyone was talking about TurboQuant. It's not perfect and when you're talking about long form reasoning, little differences can make or break the results so it is situational.

I'll read it. It could be the quants too. Some quants I try are inexplicably bad, some seem better than official (or unsloth) quants... even what should be run of the mill gguf quantization.

Re: Qwen 3.6 27B is the sweet spot for local development

#767

Post by iagooar »

bfjvibybd6cuvu6 wrote: post #656
iagooar wrote: post #624

Earlier quoted context omitted.

On paper the M4 should be roughly 1/3 of the M5, in practice it is only 1/2. With the right, optimized model like qwen3.6 35B MoE MLX you can get over 40 tok / sec on it. I run dozens of background jobs that are not time-critical on it.

What kind of jobs?

Scanning mailbox, reading and classifying emails. Scanning a knowledge base, reading and improving individual articles, reading support interactions and creating summaries, checking and researching new leads that signed up. So much that is possible.

Re: Qwen 3.6 27B is the sweet spot for local development

#768

Post by SwellJoe »

lambda wrote: post #717
girvo wrote: post #393

Earlier quoted context omitted.

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision. I do like Gemma for translation, however.

Gemma 12B? It's unique in the Gemma family, and unique among vision models. It's a novel encoder-less model...the whole model is vision. Somehow. I don't understand it, but it blows away Gemma 4 31B and Qwen 27B in my tests. It's not even close. And, is also tiny and fast, compared to those larger models, so it's better and faster and smaller. Weird combo.

Re: Qwen 3.6 27B is the sweet spot for local development

#769

Post by imrehg »

nunodonato wrote: post #627
imrehg wrote: post #612

I'm having a decently good time time with `qwen3.6-35b-a3b-mtp` (unsloth's multi-token prediction version) and and `qwen-agentworld-35b-a3b`. On a 2021 M1 Pro (32GB RAM) I can get either of them as `IQ4_NL` quantized models (the first with reduced context, around 160k; the second can do the whole 264k with RAM left over), running something like 30tokens/s. On a Framework 13 AMD AI HX370 it can use the same, but both…

what are you using agentworld for?

From the Huggingface page, it is a fine-tuned version of the Qwen3.6-35B-A3B that is my alternative, and the benchmarks seems to be better. So I'm using as a "likely some quality gain over the other model, while the performance seems pretty much the same".

I've done a couple of checks, and it seems very marginally better on some local benchmarks I'm running, but it's not super scientific evaluation.

Re: Qwen 3.6 27B is the sweet spot for local development

#770

Post by a96 »

coldtea wrote: post #363
CamperBob2 wrote: post #159

Earlier quoted context omitted.

When's the last time you shoed a horse? The reason I delegate so much of local LLM installation and administration to Claude Code is simply because there's no point learning practical things that will work completely differently in a couple of years, or in memorizing procedures that I'll forget long before I need to perform them again. No longer having to sweat all the details is a Good Thing, not a Bad Thing.

> When's the last time you shoed a horse? That skill died too, so what's your point?

It did not. There are professional horse shoers like there always were. Not in huge numbers, but there are. Domesticated horses didn't just disappear from the world.
Post reply on HN