Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.6 27B is the sweet spot for local development

quesma.com

Post reply on HN
Print view View original

301–310 of 809 posts

Re: Qwen 3.6 27B is the sweet spot for local development

#301

Post by sanderjd »

oceanplexian wrote: post #221
smt88 wrote: post #181

Earlier quoted context omitted.

Economies of scale are a fact of nature and aren’t going to be subverted in the future by even the most advanced local models

Things can get both more expensive and cheaper at scale, hence the term. For example (and relevant to AI) I can generate electricity on my roof at $0.20-25/kWh, batteries included. In California the electric utility can’t offer it cheaper than $0.30-0.50/kWh. Therefore at scale, electricity is actually more expensive. There are many such examples.

Yeah, I think the fallacy here is the conflation of scale and centralization.

Right now, there is way more scale in centralized AI than there is at the edge. But that could flip. I'd still probably put the probability that it will under 50%. But I'd also put it above zero!

Re: Qwen 3.6 27B is the sweet spot for local development

#302

Post by ricardobayes »

dofm wrote: post #279
not_kurt_godel wrote: post #267

Earlier quoted context omitted.

> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Agree having a powerful machine is really worth it in general for professionals, but strong disagree that running local LLMs has anything to do with it. It's hard enough as it is getting a good ROI on your time/money prompting/wrangling with frontier models. IMO leaning on the comparatively limited capabilities of local…

I'm not that bothered about my coding skills, which are fine, and pretty up-to-date considering I'm now an old bloke. I am bothered about building an instinctive understanding that helps me deal with my anxieties and decide whether I want to carry on with this working life or quit. I needed to do this, this way, in my own time, to put my brain back together. It has worked for me, which is why I recommend it. YMMV.

Unfortunately the local llm bunch is not the most emphatetic one in my experience: you are somehow "expected" to immediately know all this stuff and god forbid you ask the wrong question. I've never seen or felt this level of bullying and weird vibes over tools and LLM models. "My setup works for you or beat it".

Re: Qwen 3.6 27B is the sweet spot for local development

#303

Post by cpburns2009 »

Before you run and go purchase a unified memory computer (e.g., DGX Spark, Mac, Ryzen AI Max 395 / Strix Halo), be aware dense models generally run slow on these machines. Dedicated GPUs run dense models significantly better. Look for benchmarks for your prospective machine. If you really want one of these, you'll be better off running Qwen 3.6 35B or another sparse MoE model.

Re: Qwen 3.6 27B is the sweet spot for local development

#304

Post by jboss10 »

Otternonsenz wrote: post #91

Is there any hope for people that cant even run 27B parameters, Qwen3.6 or otherwise? Are there any quantized models that do well with tool calling at smaller parameter sizes? I do not have a crazy rig, a modest gaming one at that, but in trying to understand more about agents and their capabilities, I am SOL with my 16 GB of RAM and 8GB of VRAM. I can get most small, non tool calling models to perform well, but I've…

I have 8GB VRAM but 32GB RAM. Qwen 3.6 35B runs nicely.

You should look at gemma-4-26B-A4B. 16+8=24gb and Q4 is about 16GB. Not much context left, but might run.

Re: Qwen 3.6 27B is the sweet spot for local development

#306

Post by monkmartinez »

dofm wrote: post #169
wahnfrieden wrote: post #160

Earlier quoted context omitted.

Difficult... and wastefully expensive

I mean, it's a (secondhand) computer I bought for other tasks (processing very large photos, compiling large apps quickly). It's running all the time. It can also run LLMs when I want to. The rest of my life is ultra-frugal so I am relaxed about this.

My thinking is totally aligned with yours, perhaps its because I am trying to do a second act at almost 50 from blue-collar to white collar office work. I have no formal degree, but I have been hobby programming for 20 years. I have made a habit of "letting myself be available to all lessons"... the localllama group has made this journey really fun if nothing else. I have learned an ABSOLUTE ton from this era!

Re: Qwen 3.6 27B is the sweet spot for local development

#307

Post by iagooar »

oceanplexian wrote: post #207
iagooar wrote: post #199

I love my MacBook Pro M5 128GB RAM and I love qwen3.6. BUT DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise. Running any kind of sophisticated job on the very laptop you are using is just not viable. Sure you can use it in clamshell mode, but forget touching it while working with AI coding or ag…

If you want to do coding with a local LLM your best bet is a 6 year old Nvidia 3090 which is substantially more powerful than the highest end overhyped Apple product for 1/5th the price.

My problem is I won't accept anything lower than the 96GB the RTX Pro 6000 Blackwell has. My dream is a workstation with 2x Pro 6000 to run DeepSeek v4 Flash comfortably, possibly qwen 3.6 / ornith on turbo speed.

But man, I have never purchased a computer which is more expensive than a decent family car.

Re: Qwen 3.6 27B is the sweet spot for local development

#309

Post by dofm »

ricardobayes wrote: post #292
VerifiedReports wrote: post #278

Earlier quoted context omitted.

Exactly. The distinction between the various layers in "AI" systems is pretty vague to the newcomer. What is the "model" vs. the engine "running" it vs. weights? I don't recall any previous tech stack that was barfed onto the scene with so little background or reference material, going from zero to endless undefined jargon... and no primer in sight. For people who demand an understanding of their tools, it's a lot of…

For the most part you can just download LM Studio and go from there. It provides a chat interface and an easy-to-use interface to browse, load and use LLM models. The engine: it is abstracted away by LM Studio, if you want to dig deep it's llama.cpp as the runtime. Weights are the files what you download, they are the models for practical purposes.

I definitely would recommend LM Studio as a learning environment, because it surfaces a bunch of things in relatively clear-minded ways. I am very grateful for it.

Re: Qwen 3.6 27B is the sweet spot for local development

#310

Post by jboss10 »

MangoCoffee wrote: post #218

Running LLMs locally for development doesn’t make sense to me. The hardware gets outdated in just a few years. Even hyperscalers replace their GPUs faster than they can buy them, plus the cost of running it locally, isn’t cheap. the cost saving just ain't there.

Qwen 3.6 35B runs on 32GB with a 1080. That GPU is from 2017.
Post reply on HN