Qwen 3.6 27B is the sweet spot for local development

Post by 4chandaily » Mon, Jun 29, 2026, 7:15 PM UTC

stymaar wrote: post #161

boutell wrote: post #123

Earlier quoted context omitted.

That 3090 is going to burn 750W and it will still cap you at a 4 bit quant and ~48K context. Here's someone who worked through it: https://github.com/noonghunna/qwen36-27b-single-3090 Flies though (50-70tps is impressive for a model this smart) I went through roughly the same process to get it working on my M2 Macbook Pro... at awful speeds of course, since models like this one are mostly bound by memory bandwidth.

> That 3090 is going to burn 750W The 3090's TPD is 350W, but given that LLM's token generation isn't compute bound, people usually undervolt these cards to reduce power consumption. IIRC you can get as low as 200-250W without any degradation. Caveat these figures are without speculative decoding and at batch size =1.

Post by verdverm » Mon, Jun 29, 2026, 7:17 PM UTC

dofm wrote: post #171

CamperBob2 wrote: post #159

Earlier quoted context omitted.

When's the last time you shoed a horse? The reason I delegate so much of local LLM installation and administration to Claude Code is simply because there's no point learning practical things that will work completely differently in a couple of years, or in memorizing procedures that I'll forget long before I need to perform them again. No longer having to sweat all the details is a Good Thing, not a Bad Thing.

I am not sure I disagree, and I certainly don't mean to disagree very fervently. But I think if you want to really learn to ride well, understand horses well, there might be some benefit in learning how to shoe a horse. At some level it should never only be someone else's job.

Post by fluoridation » Mon, Jun 29, 2026, 7:18 PM UTC

Otternonsenz wrote: post #180

fluoridation wrote: post #140

Earlier quoted context omitted.

I think at 16 GB you'd struggle to run the regular development tools nowadays, forget about any interesting inference.

Fully agreed, and my hope is as open models grow and change, that getting some amount of this working on Pro-sumer hardware will be more attainable. But certainly seems like we are a few years away from that, sadly. Am I also screwed in being able to train my own small model or adjust another one with such a non-workhorse PC?

Post by nozzlegear » Mon, Jun 29, 2026, 7:19 PM UTC

beastman82 wrote: post #43

FWIW I'm running gemma4 31b on my 5090 and it's pretty great as well. QAT, MTP, 128k context. I liked Qwen 3.6 27b too, it just seems that Gemma4 is a bit underrated.

Post by verdverm » Mon, Jun 29, 2026, 7:20 PM UTC

sosodev wrote: post #71

onion2k wrote: post #20

None of the examples reflect 'real work', at least not what I'd consider real work. Being able to nail a zero-shot greenfield project is relatively easy even for a small model. There's not much context to build up and it can fall back to similar examples in the training data easily. So long as you're not asking it to invent something wholly new it'll probably manage. The real test is whether or not it can work with y…

In my experience, even with basic project concepts the small models struggle to spin up greenfield stuff. There's just too many decisions to be made and they're not good at that. Modifying existing code is way easier if you don't expect it to be smart about it. Don't say "add X feature" and let it explore the codebase and build its own understanding. Point it at the relevant files and say "the goal is to add X featur…