Running local models is good now • Hacker News Board

Re: Running local models is good now

#351

Post by aftbit » Tue, Jun 16, 2026, 8:06 PM UTC

jtbaker wrote: post #346

trueno wrote: post #326

Earlier quoted context omitted.

someone just put this on my radar yesterday, im about to try this today. how's your experience with it? me thinks there's a lot of optimization strats we're currently leaving on the table just because the amount of things to explore and test are so expansive. but this one is super interesting targeting metal primarily and zeroing in on one model. instead of a one size fits all llama.cpp im very interested to see if t…

I'm on the verge of cancelling my anthropic $20 plan since it's come out. On an M5 Max 128GB, hooked up to the pi.dev harness, I get in the neighborhood of 400-450tps prefill and 30-35tps generation. It is imminently usable and at times feels more stable than my previous CC setup. Occasionally there are things it struggles with that I will bounce back over to CC for, but it is highly usable. The future is bright for…

That's DS4 Flash right? How does it feel in intelligence and speed compared to DS4 Flash hosted by Deepseek themselves or another API provider? I've been using API DS4 Flash for a lot of personal projects and have been quite impressed. I've spent $1 on building ~10 toy projects and gotten them all to work within the bounds of what I wanted without having to do much besides guide the model away from dumb loops.

Re: Running local models is good now

#352

Post by mft_ » Tue, Jun 16, 2026, 8:07 PM UTC

jayGlow wrote: post #237

trilogic wrote: post #142

Earlier quoted context omitted.

I also confirm that local inference is on par with proprietary cloud services (with a bit of local setup, simple agents.md and some utils skills). This local models come with tools, that's mind blowing, considering that some months ago we had to .md tools ourselves. What makes a model worth even more is "Memory". We implemented that long ago. Last time I used proprietary services was 3 months ago, don´t really need i…

what are you using for memory with your local models? is there a specific harness you would recommend for local agents?

I’m using Hermes at the moment - it comes with lots of tools already baked in for the agent to use - for example web and browser access just worked, rather than having to mess around loads with config scripts and plugins.

I’ve also tried OpenCode (similar but a bit less so) and Pi (fast but you have to add lots of features yourself which is a bit of a pain). Claude Code can also be pointed at a local model and works, but the default system prompt is huge. (~140k of text when I extracted mine, IIRC.)

Re: Running local models is good now

#353

Post by Mr_Eri_Atlov » Tue, Jun 16, 2026, 8:07 PM UTC

I think this is a pivotal moment for LLMs.

Gemma 4 and Qwen3.6 27B aren't perfect, yet they are such a step forward from the previous generation that it's both feasible to get stuff done locally with patience and very likely that future releases will subvert cloud capabilities entirely.

Plus, they have definite reliability advantages over cloud models that can be wiped out by a government order or lobotomized to handle traffic surges.

Re: Running local models is good now

#354

Post by aftbit » Tue, Jun 16, 2026, 8:08 PM UTC

EagnaIonat wrote: post #262

aftbit wrote: post #165

Earlier quoted context omitted.

IMO running local models "well" still requires an expensive hardware investment. You really want 96GB of VRAM on a modern Blackwell arch to run these models with decent KV cache. Trying to run them on a unified memory Mac, an AI Max AMD processor, or a DGX Spark-alike is really just asking for trouble. Prefill kills perf. If you throw the right GPUs at the problem, they become much better - but still not quite in the…

Depends what you need the model to do. The recent granite4.1:3b just takes 2GB of memory and is fast. Results are pretty good and support tool calling. Barely a squeak out of the Mac laptop. Even faster with the MLX builds. Then when I need more heavy lifting I fire up a larger model. IMHO the issue isn't the models. I've had OpenClaw give the same results as Claude using open models locally. Slower but does the job.…

Yeah it 100% depends what you want the model to do. Some tasks, like extraction, summarization, or simple tool calling (e.g. "turn on my desk lamp") are very doable with tiny models. Others, like coding or more advanced agentic workflows can demand much more powerful models. I was thinking from the lens of coding or running _big_ data extraction pipelines (think ~8 billion pages).

Re: Running local models is good now

#355

Post by amdivia » Tue, Jun 16, 2026, 8:11 PM UTC

Stagnant wrote: post #337

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

I've been using unsloth/gemma-4-31B-it-qat-GGUF daily for various small parsing and programming tasks using opencode and llama-server's front end. The past couple of weeks have made a big difference after google released the QAT variant and llama.cpp got support for MTP which means it is possible to now get 60-80 Tok/s with RTX 4090. The model fits in VRAM comfortably enough to keep it loaded even while browsing and…

110+ Tok/s as another data point on the RTX 5090 (Gemma 4 31B QAT + MTP at UD-Q4_K_XL) (at peak used 27 GB of vram)

The real lovely thing was getting 300+ Tok/s (Gemma 4 26B QAT + MTP at UD-Q4_K_XL) (at peak, I think I saw vram usage reach 21 GB of vram)

Re: Running local models is good now

#356

Post by frereubu » Tue, Jun 16, 2026, 8:11 PM UTC

epistasis wrote: post #85

hypfer wrote: post #15

After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…

> talking just way too much OMG this is such an annoying property, just shut the hell up please, and be concise. I suspect that this is an artifact of the thinking property, but please just summarize the thinking process far more concisely, where a single sentence answer is more than sufficient the frontier models seem devoted to going on to a minimum of 5 paragraphs and offering 3-5 new directions. And requests to p…

[dead]

Re: Running local models is good now

#357

Post by papersail » Tue, Jun 16, 2026, 8:12 PM UTC

pornel wrote: post #344

[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…

I had similar doubts. I think expectations differ because the workload differs. For small scripts, glue code, or simple CRUD changes, smaller models such as Qwen3.6-27B can work wonders than they do on a larger, messier code base.

Re: Running local models is good now

#358

Post by mohamedkoubaa » Tue, Jun 16, 2026, 8:13 PM UTC

I wonder when a cheaper consumer grade inference chip will hit the market. The general purpose GPUs have much more silicon and complex firmware than what's strictly needed for inference

Re: Running local models is good now

#359

Post by bayshark » Tue, Jun 16, 2026, 8:15 PM UTC

Hey everyone, made a local LLM, configured for Home Assistant called Selora AI.

Specs: qwen3_17b_base.Q6_K.gguf selora-v047-answer.f16.gguf selora-v047-automation.f16.gguf selora-v047-clarification.f16.gguf selora-v047-command.f16.gguf

The full base model and LoRA adapters are only 3.5GB

Capabilities include configuring for smart home setup to help with answers, clarifications, commands, and creating automations in Home Assistant. The models with the LoRA adapters were made with lean scripted data made specifically for Home Assistant. A lot of work was put into this, feel free to give it a try and happy for any feedback!

https://huggingface.co/selorahomes/Selora-AI

Re: Running local models is good now

#360

Post by famouswaffles » Tue, Jun 16, 2026, 8:15 PM UTC

embedding-shape wrote: post #332

famouswaffles wrote: post #317

Earlier quoted context omitted.

Difficulty of scaling is not the only issue. Nobody is going to be particularly invested in scaling an architecture that has: - consistently proven behind their auto-regressive counterparts in quality. Look at the dgemma benchmarks - pretty steep dropoffs and the more difficult the benchmark the worse the dropoff. That's not a good look and it's not like its some artifact of google's release. Every dllm is like this.…

> - And whose inference benefits are negated at scale. Transformers are still cheaper if you want to serve lots of users. But my entire point is about the reverse of this, the context of what I bring up is in single-user scenarios, which is where these diffusion models really make a large difference in performance. Sure, I agree it's not a good fit for every single use case out there, everywhere. But after starting t…

I don't think you're really getting the point I'm trying to make. Everyone training llms regularly cares about serving users at scale and quality per compute invested. It's not just about OpenAI or Anthropic or Google. Qwen, Deepseek, Moonshot, whatever. They all care about it very much and basically can't afford to take a step back in those areas.

Since training models is currently a very expensive procedure, diffusion llms are destined to be relegated to the occasional research artifact at best. As things stand, making a serious commitment to them is basically the equivalent of throwing money into a fire pit and things are expensive enough as is.

Alternate Architectures that do a much better job matching transformers in quality have basically gone nowhere but you expect one that is basically worse in every way the labs care about won't ? I'm not trying to 'dismiss' dllms. I'm interested in them for the same reason you are. I'm just stating the factors at play plainly.