Running local models is good now • Hacker News Board

Re: Running local models is good now

#231

Post by ridiculous_leke » Tue, Jun 16, 2026, 5:43 PM UTC

greenavocado wrote: post #143

zozbot234 wrote: post #116

Earlier quoted context omitted.

Maybe we shouldn't be running these models on laptops with their thermally constrained form factor, and we shouldn't expect quick inference on a par with a large cloud-based platform either, at least not for near-SOTA model quality. It's still worth it to avoid becoming massively reliant on centralized services.

I have a 5070 12 GB laptop GPU and can hit 72 tokens per second in the first couple thousand tokens before dropping to mid-high 50s after about 15k context. This setup is extremely optimized down to the last flag. Changing any param above the temp flag craters performance. I don't have enough system RAM to properly handle the large context windows so I don't use local models. # 1,257 tokens 17s 72.18 t/s $env:CUDA_DE…

Can you comment on the quality and accuracy of it? People have managed to run Gemma 26b without GPU on old CPUs but I don't think quality is anywhere close to what Gemma 12b offers.

Re: Running local models is good now

#232

Post by nickthegreek » Tue, Jun 16, 2026, 5:44 PM UTC

Barbing wrote: post #126

iagooar wrote: post #115

I love running two models locally: qwen3.6 27B 8bit (dense) and qwen3.6 35B 4bit (MoE). The 27B is the smarter, more reliable one - but it is slower. The 35B is faster, still very smart but below 27B, a bit less reliable. The reason is the MoE - Mixture of Experts architecture, which only activates a subset of parameters, making the model much much faster. I run the 27B on a MacBook Pro M5 Max + 40 GPU cores + 128GB…

Did you get a Brave search API key or something for that “Hermes”?

I have my mine setup with a searxng instance I run in a docker. Works great and costs zero.

Re: Running local models is good now

#233

Post by FuriouslyAdrift » Tue, Jun 16, 2026, 5:46 PM UTC

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Kimi 2.6 or 2.8 is what we are playing with locally. They need 512GB to 1TB to run with full capabilities so that's not exactly "desktop"

Our GPU computer server cost $110k.

Re: Running local models is good now

#234

Post by greenavocado » Tue, Jun 16, 2026, 5:48 PM UTC

mattmanser wrote: post #192

greenavocado wrote: post #143

Earlier quoted context omitted.

I have a 5070 12 GB laptop GPU and can hit 72 tokens per second in the first couple thousand tokens before dropping to mid-high 50s after about 15k context. This setup is extremely optimized down to the last flag. Changing any param above the temp flag craters performance. I don't have enough system RAM to properly handle the large context windows so I don't use local models. # 1,257 tokens 17s 72.18 t/s $env:CUDA_DE…

That's a quant 4 which the thread OP specifically called out as rubbish. The Q4_K_XL bit for those not in the know.

I typically find myself using a context of between 150-500k with GPT models so local models are simply not enough and I stopped using them.

Re: Running local models is good now

#235

Post by stymaar » Tue, Jun 16, 2026, 5:48 PM UTC

mattmanser wrote: post #192

greenavocado wrote: post #143

Earlier quoted context omitted.

I have a 5070 12 GB laptop GPU and can hit 72 tokens per second in the first couple thousand tokens before dropping to mid-high 50s after about 15k context. This setup is extremely optimized down to the last flag. Changing any param above the temp flag craters performance. I don't have enough system RAM to properly handle the large context windows so I don't use local models. # 1,257 tokens 17s 72.18 t/s $env:CUDA_DE…

That's a quant 4 which the thread OP specifically called out as rubbish. The Q4_K_XL bit for those not in the know.

Anyone calling Qwen3.6-35B-A3B-Q4_K_XL “rubish” has no idea what they are talking about.

Re: Running local models is good now

#236

Post by pjmlp » Tue, Jun 16, 2026, 5:49 PM UTC

Only if blessed with enough RAM and disk space,

> 64 GB RAM and 1TB storage

Ah ok, not something regular joe and jane happen to have lying around at home.

Additionally the whole configuration is still very much low level, bunch of CLI commands, and if the model doesn't fit for the task at hand, it starts allucinating, generating gibberish, whatever.

Re: Running local models is good now

#237

Post by jayGlow » Tue, Jun 16, 2026, 5:49 PM UTC

trilogic wrote: post #142

ggerganov wrote: post #83

Earlier quoted context omitted.

I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding tasks. Over the last month and a half I've been using it almost daily, either on my M2 Ultra or on my RTX 5090 box. I use it for small mundane tasks at ggml-org [0] - nothing really impressive, but definitely a helpful tool for a maintainer…

I also confirm that local inference is on par with proprietary cloud services (with a bit of local setup, simple agents.md and some utils skills). This local models come with tools, that's mind blowing, considering that some months ago we had to .md tools ourselves. What makes a model worth even more is "Memory". We implemented that long ago. Last time I used proprietary services was 3 months ago, don´t really need i…

what are you using for memory with your local models? is there a specific harness you would recommend for local agents?

Re: Running local models is good now

#238

Post by aamoscodes » Tue, Jun 16, 2026, 5:50 PM UTC

rapind wrote: post #226

saghm wrote: post #136

Earlier quoted context omitted.

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…

> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…

You can pay, and also use deepseek-v4-flash. OpenRouter even lets you "block" or limit your usage to providers that don't train on data. Since the weights are open, other companies are already serving the model on non-DeepSeek owned hardware: https://openrouter.ai/deepseek/deepseek-v4-flash

Re: Running local models is good now

#239

Post by bthornbury » Tue, Jun 16, 2026, 5:50 PM UTC

the qwopus 27b model is good for grunt work style tasks, even across multiple files. Piping a bunch of things through, small factoring changes, stuff that just takes time to type out.

I wouldn't rely on it for large stuff like codex though. I haven't tried out deepseek/kimi, if we could run those locally it would be great.

Re: Running local models is good now

#240

Post by andix » Tue, Jun 16, 2026, 5:50 PM UTC

Because I've seen too many people spending a lot of money on expensive hardware, without really using it in the end:

Most of those models are also available via Openrouter and many other platforms. Dirt cheap, and much faster than on consumer GPUs. Perfect to try and compare the different options.