Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

791–800 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#791

Post by SwellJoe »

automajicly wrote: post #785
SwellJoe wrote: post #760

Earlier quoted context omitted.

I don't think "large projects" is realistic with a model that fits in ~8GB (I'm assuming you run stuff other than the model). And, Gemma 4 12B QAT at 4-bits is surely the smartest model in its size, but it shines at vision tasks rather than agentic tasks (though it is a good tool user and can do stuff like research, it's obviously not aimed at code). You can almost always find free models on OpenRouter. Google AI Stu…

WOW. That was alot of models. I will be sure to look into those. What caught my eye was the North Mini Code. How is that at reason and context? I guess I will just go see. Thank you for the insight. I am building and learning at the same time, and I really need a model that wont chew up my poor little M1, but at the same time will understand my direction, my context, and also my explanation of need and then be ...' o…

North Mini Code is on par with the similar-sized MoE Qwen 3.6 and Gemma 4. It benches a little better than Gemma 4 26b a4b and a little worse than Qwen 3.6 35b a3b for most coding-related and agentic tasks. For my use case, or at least the use case I've tested, it is a little worse than both for surfacing security bugs, but all of the MoE models at this size are pretty bad at finding security bugs (they hallucinate a lot of false positives, which degrades the value of their real bugs dramatically).

Poolside Laguna XS.2 is another in this category (30B-35B MoE, reasonably competitive on coding related benchmarks). Also free on OpenRouter. But, also, it's bigger brother, Laguna M.1 (225B A23B MoE), is also free on OpenRouter last time I checked. Worth a look. https://openrouter.ai/poolside/laguna-m.1:free

The good thing about all the 30-ish MoE models is you can run them on any 32GB GPU, even old ones, at a very comfortable speed. A 24GB GPU can run the 4-bit quantizations if you use a quantized K/V cache. That's why there are so many of them. It's the sweet spot for "good enough to be useful for some coding tasks, small enough to fit on the GPU a lot of people have".

The cheapest not rate-limited options that are actually pretty competitive with the frontiers are from DeepSeek and MiMo. DeepSeek V4 Flash and Pro are extremely cheap, their caching is the best in the industry (and their cached tokens are even cheaper), and Reasonix is an excellent CLI harness that is designed around maximizing cacheability of DeepSeek models, specifically. I used it for an hour last night and spent something like three cents. MiMo has token plans that are a pretty good deal (though confusing...the token plan buys credits, and credits are not a whole token, so you get billions of credits on the token plan for a few bucks, but it chews through it at a rate faster than 1 credit per token). But, DeepSeek V4 Pro is a consistently better model than MiMo.

Re: Qwen 3.6 27B is the sweet spot for local development

#792

Post by QuercusMax »

dofm wrote: post #790
QuercusMax wrote: post #784

Earlier quoted context omitted.

I've been using pi-coder the couple days with Qwen, and its minimal prompt works MUCH better than OpenCode's bloated prompt.

I was going to reply with my answer but I tested it again and realised that actually in some cases the custom prompt is somehow being ignored, so I don’t truly know how good it actually is in opencode. Poking around in the convoluted opencode source or the overstuffed github issues to try to make sense of exactly what is happening has convinced me to switch to pi as well, simply to have a thing I can more fully under…

That was basically the conclusion I came to after I spent a couple hours trying to puzzle my way through opencode's settings - extremely opaque and hard to tell what's happening. Pi is simpler, and i figure if I really need a better system prompt, that's literally what AGENTS.md is for, right?

Re: Qwen 3.6 27B is the sweet spot for local development

#793

Post by jasonjmcghee »

jasonjmcghee wrote: post #579
iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always…

I usually get mlx versions of models and didn't realize I was using non-mlx - misattributing the performance boost to speculative decoding.

With dflash-mlx library, with dflash disabled 3.6-35b-a3b mlx model I'm getting ~60 t/s on low power and >100 t/s on high power.

Compared with previous message being official qwen huggingface release (non-mlx) using lm studio.

Re: Qwen 3.6 27B is the sweet spot for local development

#794

Post by jasonjmcghee »

anon373839 wrote: post #584
jasonjmcghee wrote: post #579

Earlier quoted context omitted.

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always…

Can you mention what inference stack you're using? I've tried MTP several times with that model and it always seems to significantly cut my token generation speed from ~60 tokens/sec to ~40 (M3 Max).

(see above reply to myself) I misattributed the gain from dflash - it was dflash-mlx library + mlx model, not dflash itself giving me the speedup.

Re: Qwen 3.6 27B is the sweet spot for local development

#795

Post by drnick1 »

tedivm wrote: post #754
drnick1 wrote: post #743

Earlier quoted context omitted.

How useful is the second 3090 in this setup? I run the 5-bit quantized model on a single 3090. Does the second 3090 allow you to use the full precision model instead or a less aggressive quantization by splitting the layers? What about running the 35B model instead?

More memory means less aggressive quantization, more concurrent requests, and larger context windows. I also get a boost in tokens per second (not double, about 1.5x compared to a single GPU). The 35B model is an MoE (mixture of experts), which uses only a subset of parameters at a time. The 27b one is slower but has way better performance.

Nice, I wish I could find another cheap 3090 somewhere.

Re: Qwen 3.6 27B is the sweet spot for local development

#796

Post by dang »

spiralcoaster wrote: post #427
hollowturtle wrote: post #387

Earlier quoted context omitted.

In what era spinning up a PoC required a week of work? Especially on the web. I've been a developer for roughly 20 years and that has never been the case, to the point that I believe people impressed by LLMs are the same who had a very low productivity. Today we have game jams as short as 3 days and talented people are able to produce very good PoC, with some almost complete!

[flagged]

We've banned this account for attacking other users and ignoring our request to stop, as well as frequently breaking the site guidelines in other ways. Not cool, and not allowed here.

If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.

Re: Qwen 3.6 27B is the sweet spot for local development

#797

Post by lambda »

SwellJoe wrote: post #768
lambda wrote: post #717

Earlier quoted context omitted.

I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision. I do like Gemma for translation, however.

Gemma 12B? It's unique in the Gemma family, and unique among vision models. It's a novel encoder-less model...the whole model is vision. Somehow. I don't understand it, but it blows away Gemma 4 31B and Qwen 27B in my tests. It's not even close. And, is also tiny and fast, compared to those larger models, so it's better and faster and smaller. Weird combo.

Tried it out. I'm compring against Qwen 3.5 122B-A10B, so a much larger model. It gets some correct, but Qwen 3.5 122B-A10B has done much better. Gemma 4 12B even hallucinated some species in trying to identify a plant, and the other guesses it made weren't all that close, while Qwen 3.5 122B-A10B got it right on the first try.

12B did get one right that 31B got wrong. I'd have to do a much more thorough eval to really compare, just a few anecdotal observations and it's kind of hard to really distinguish, but from the samples I've seen, Qwen 3.5 122B-A10B is doing much better at this task.

The 12B architecture definitely is interesting, and it may punch above its weight due to this (though again, would really need to do proper evals to compare). But of the models I've tried, Qwen3.5 122B-A10B really seems like the best for this kind of task.

Re: Qwen 3.6 27B is the sweet spot for local development

#798

Post by ljosifov »

kroaton wrote: post #480
ljosifov wrote: post #322

Earlier quoted context omitted.

True - they are workhorses. Not super bright, but good enough for lots of everyday tasks. I've found sweet spot to be turning thinking off, as it adds small or no value, while increasing the token count and waiting time. Last 27B I used was https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-GGUF - specifically post-train adapted a bit to run with thinking off. I saw today the 35B-A3B MoE from the same HF acc is out,…

Please don't use that garbage. Just use the base Qwen models or Nex/Orinth, as those are the only properly post-trained finetunes. The Qwopus models are marketing.

I usually doubt the 'small dataset tuned' variants. B/c ages ago (in the NN prehistory) I've done some NN training, and appreciate how hard it is to improve in general, and how easy it is to ruin a model in general while targeting a small dataset (LoRA-s are ok, that's different). That model/quant was the most recent one I was trying. But could not really use any of them, as the combo model + llama-server ground to a halt even at small context depth sizes on the amd gpu.

Yesterday I finally found a good combo! So writing this for the benefit for anyone that may have the same h/w. Got around to /GOAL search for something better for the h/w (amd 7900xtx), and pi agent found a new best that actually seems it will be useful for real. As the 40 tok/s speed starts dropping only at 260K context depth?? Served by hipfire from this repo https://github.com/Kaden-Schutt/hipfire, that worked the best got on llama-benchy:

  | Context | Wall Time | PP (t/s) | TG (t/s) |
  |:-------:|:---------:|:--------:|:--------:|
  |   1,024 |   17.3s.  |   660    |    40.8  |
  |   8,192 |   29.3s   |   606    |    40.8  |
  |  32,768 |   70.5s   |   599    |    40.7  |
  |  65,536 |  126.5s   |   594    |    40.6  |
  | 131,072 |  235.5s   |   591    |    39.0  |
  | 196,608 |  349.0s   |   591    |    40.2  |
  | 260,000 |  451.3s   |   594    |    37.2  |
This is - hipfire daemon dflash_mode auto, serve qwen3.6:27b --kv-mode asym2. Models:

  hipfire pull qwen3.6:27b           # 14 GB MQ4 quant, qwen3.6-27b.mq4
  hipfire pull qwen3.6:27b-draft     # 0.92 GB DFlash draft, qwen36-27b-dflash-mq4.hfq

Re: Qwen 3.6 27B is the sweet spot for local development

#799

Post by btbuildem »

whichquestion wrote: post #475
btbuildem wrote: post #389

Earlier quoted context omitted.

I think it's silly to go for a laptop form factor. Last fall I put together a workstation with two second-hand 3090s in it (paid $850CDN each, now the best I can find is $1200). With 48GB VRAM it's reasonable - and I've been using Qwen 3.6 27B for various tasks around building KGs from text corpora / reasoning about them. I've ran comparisons against everything that's available on OpenRouter (well, as of few weeks ag…

How much electricity does running your local models take?

I throttle the GPUs to ~280W each (my cooling solution is insufficient to run them full tilt), and so at peak usage I show a ~1200VA draw. Electricity is relatively cheap here, so the main disadvantage of the at-home setup is having the equivalent of a small space heater running in the summer.

Re: Qwen 3.6 27B is the sweet spot for local development

#800

Post by automajicly »

SwellJoe wrote: post #791
automajicly wrote: post #785

Earlier quoted context omitted.

WOW. That was alot of models. I will be sure to look into those. What caught my eye was the North Mini Code. How is that at reason and context? I guess I will just go see. Thank you for the insight. I am building and learning at the same time, and I really need a model that wont chew up my poor little M1, but at the same time will understand my direction, my context, and also my explanation of need and then be ...' o…

North Mini Code is on par with the similar-sized MoE Qwen 3.6 and Gemma 4. It benches a little better than Gemma 4 26b a4b and a little worse than Qwen 3.6 35b a3b for most coding-related and agentic tasks. For my use case, or at least the use case I've tested, it is a little worse than both for surfacing security bugs, but all of the MoE models at this size are pretty bad at finding security bugs (they hallucinate a…

I appreciate the insight. I might look at these a bit later... because I am swamped with this new build Im putting together. I havent yet found my so-called 'sweet spot' yet for larger tasks. I checked out the gemma4 variant the one or two times just for contextual reasoning and smaller build capabilities, but Ive only been testing single use isolated tool tests with my mcp pointed at it lately to ensure tools I've installed actually work before I incorporate them into the mcp server. No plug or anything, but just so you can get a feel of what I'm doing, I initially set out to learn coding. Then I fell into trying to build my own local model to help teach me coding (and anything else related) because tutorials and all these fancy webinars and such just put me to sleep. Once I discovered all of this AI stuff...? My interest in cybersecurity just skyrocketed. So....no life story or anything) ha... I started to build this mcp, pointed at LM Studio with qwen loaded as my first one. As I began to familiarise more I realised...I need a model that is free, local, will understand context, reasoning, can code, de-bug,vuln discovery- etc. etc. All of this sent me on this whole cybersecurity rabbit hole - as it does with any tech stuff, and now along with my need for a model to teach me coding(Python) I wanted a model to also HELP me build ITSELF basically. Because so far, Ive used good old Claude- Sonnet-5 straight from the mobile app for said task. Dont laugh. And I show the build,and upgrades of the mcp to GEMMA (initially qwen) and get fedback against the Cluade assisted builds. So I apologise for ruining your day, but... thats where Im headed. I need a model that will vibe with me, build with me, reason, comprehend context and also... not chew up my m1 16gb, or token usage. Well, if its free I suppose tokens arent a thing but they do sort of matter still. Thanks again. Your information is much appreciated and invaluable.
Post reply on HN