Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

691–700 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#691

Post by _tyiueojdfg4 »

My personal experience below:

I ran into some small problems with codex during setup and, for a few reasons, did not want to set up a cli shell with them at the time. Since I was not doing anything really serious, but just exploring a half-baked idea for an android app, I ran qwen in lms and connected it to android studio.

None of the mini projects that I have attempted ( more granular call control, silly html scrolling game, music play app ) were one shots despite carefully preparing the prompt ahead of time. Admittedly, some of it may have something to do with android studio, but I did not try it with google account yet. All took between an hour to four to generate ( prep, initial run, testing, iteration and so on ).

If it helps, miniforum AI MAX 395. I am not saying it is bad. Quite the opposite, but you want to be aware of the limitations though and plan around those.

Re: Qwen 3.6 27B is the sweet spot for local development

#692

Post by Tenoke »

asimovDev wrote: post #665
Tenoke wrote: post #639

Earlier quoted context omitted.

>How many times has an Apple OS wiped your hard drive or otherwise been completely borked from a forced update I use Windows and this has never happened to me. I have had Macbooks I cant open to fix/replace something trivial while I can replace any part easily on a Windows PC/laptop though.

>Windows PC/laptop though. needs to be noted that it's increasingly uncommon to be able to do so. for desktops you have to build everything yourself - prebuilds (either gaming or workstations) have proprietary PSU and motherboards (in case of workstations, sometimes CPU is bound to the motherboard / manufacturer, for example Threadrippers). Windows laptops now often come with soldered RAM and soon will probably be wi…

Well my PC is built on my own and I change parts every year or two. My laptop is a Thinkpad from last year and Im pretty sure I can easily open it to replace something just like my last one.

Re: Qwen 3.6 27B is the sweet spot for local development

#693

Post by sfifs »

brandall10 wrote: post #433
ljosifov wrote: post #255

Running 27B dense model on M5 128GB is ok, but one can do better. On M5 128GB one can make use of the ram and use sparse MoE. For example, DeepSeek-V4-Flash will fit, served by DwarfStar ( https://github.com/antirez/ds4 ). One will probably improve 2x the token/sec speed, given DS4F 13B activated params in the MoE are ~1/2 of the ~27B of the dense Qwen. 27B Of the Qwen fit even on a cheaper 24GB card, e.g. amd 7900xt…

This is discussed in the article: "My personal impression is that within these quantizations Qwen 3.6 27B is as good as (or maybe slightly better than) DwarfStar4. Though, I won’t be surprised if for longer context projects DS4 has an edge."

Used both. DeepSeek-4 Flash Q2 - last 6 layers Q4 quant with DwarfStar which just about fits in 128Gb is definitely superior IMO - my contexts tend to run typically 50-100k. Throughput tends to be about 12-13k tok/sec - just about acceptable.

Re: Qwen 3.6 27B is the sweet spot for local development

#694

Post by kelchm »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

This -- with the M5 Max MBP is running flat out, you'll go from full battery to empty in under two hours.

While it is wild to have this much power in a take-it-anywhere laptop form factor, I sort of regret not just going for a Mac Studio + base M5 MBP.

Re: Qwen 3.6 27B is the sweet spot for local development

#695

Post by blagui »

iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

So the sweet spot for dev in 2026 is 64k context windows? Are we back in 2024?

As more context will degrade a lot the t/s. On top this is 1 slot.

If you use sub agents the kv cache will be invalidated with colliding request and make it even slower.

So the in real world 256k (the max qwen offer) and using 3-4 slots the numbers are very different.

This is the major issue with so many postes over local models not benchmarking real world use. Real context and not taking this in context.

If you use 1 slot the issue, you loose the ability of using sub agents when exploring and all end up in the main agent context overloading it, triggering compactation and oh boy with 64k context that compecation will be an endless loop.

What tasks you would really be able to do with 64k context 1 agent? For sure so quick edits but not complex planning where you need to ingest a lot files and end up loosing 80% of the ingested files to compactation.

Re: Qwen 3.6 27B is the sweet spot for local development

#696

Post by jingpostmedia »

XCSme wrote: post #424

Considering the cloud version, all three models compared in the article (Qwen 3.6 35BA3b, 3.6 27B and DeepSeek V4 Flash), have very similar performance[0], BUT on cloud, for some reason DeepSeek V4 Flash is 10-20x cheaper than the Qwen models. If Qwen models are so much easier to run, why are the providers charging more than V4 Flash? [0]: https://aibenchy.com/compare/qwen-qwen3-6-35b-a3b-medium/qwe... <-- compare ho…

[flagged]

Re: Qwen 3.6 27B is the sweet spot for local development

#697

Post by Saris »

myzek wrote: post #676
Saris wrote: post #500

Earlier quoted context omitted.

Even with deepseek v4 flash I burned though $5 in credits in a day just playing around with Hermes, and qwen 3.6 35B is significantly more expensive. I can run qwen 3.6 35B on my gaming PC at around 50 tok/s and other than power cost of a tiny bit extra per month, it's hardware I already owned from years ago. I'm not really sure why qwen 3.6 35B is so expensive on openrouter, it seems abnormally high for what hardwar…

How do you run 35B on a gaming PC? I'm trying to go the same route, but I have a 5070Ti with only 16GB VRAM (I bought it for gaming) and I'm not sure how to run anything decent on it. I have 64 GB RAM if that matters

I run it on a 12GB 4070 with 32GB system RAM. 35B A4B means only part of the model is active at a time so it takes a lot less VRAM than a dense 35B model would.

The main thing in LM studio (or whatever software you use, assuming it has fairly up to date stuff and exposes the toggles) is to offload MoE layers to the CPU, and use K/V cache quantization at Q8_0 or Q4_0.

Since you have more VRAM than I do, you could probably get away with MoE offload of like 15-20 so some remains on the GPU.

Just make sure GPU offload is turned all the way up. And I use 64k context size, although with 16GB VRAM you can probably do more.

You can find the best performance spot by playing with MoE offload until you find the number that gives the highest tok/s on your hardware.

Re: Qwen 3.6 27B is the sweet spot for local development

#698

Post by jwr »

doodlesdev wrote: post #83

I feel like I'm going insane seeing people buy these 128gb MBP for thousands of dollars to run models that are objectively much worse than SOTA and spending so much more. The amount spent on a 128gb M5 MAX can buy you a damned new car here. What the hell am I missing? Are developers in other countries living in such different worlds? (I'm aware the price is, in absolute terms, more expensive where I live compared to…

Well, I can tell you how my thinking goes: 1) I don't buy my computer just to run LLMs and there are many scenarios where I benefit from both a decent GPU and from a large amount of RAM, 2) I run a solo-founder business which owns exactly one computer in the entire company so it might as well be a good one, and 3) I don't need a new car, so comparing pricing this way is irrelevant.

In other words, yes, buying this kind of machine only to run an LLM locally doesn't make sense, because local LLMs generally still suck for serious programming work (they work great for spam filtering though!). But more generally this machine makes sense for a lot of people.

Re: Qwen 3.6 27B is the sweet spot for local development

#700

Post by ctkhn »

jaggederest wrote: post #514
LeifCarrotson wrote: post #432

Earlier quoted context omitted.

I've also been running Qwen 3.6 35B A3b on my Windows laptop (64 GB RAM, a 4GB GPU) and it's at least tolerable. It's not fast - a few tokens per second, slower than reading speed - but I can give it a task and come back later. That was a $600 laptop off eBay a few years ago, not a $6,000 machine. Are these unified memory Macs and giant 24GB desktop GPUs achieving dozens or hundreds of tokens per second commensurate…

35b A3b runs ~100 tokens a second on the best M5 Max gpu setup.

I got around 50-60 on my m3 max so 100tps seems very realistic for 2 gens later of chip and double the ram
Post reply on HN