Running local models is good now

Post by [deleted] » Tue, Jun 16, 2026, 5:04 PM UTC

ggerganov wrote: post #83

hypfer wrote: post #15

After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…

I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding tasks. Over the last month and a half I've been using it almost daily, either on my M2 Ultra or on my RTX 5090 box. I use it for small mundane tasks at ggml-org [0] - nothing really impressive, but definitely a helpful tool for a maintainer…

Post by abalashov » Tue, Jun 16, 2026, 5:06 PM UTC

Post by dstryr » Tue, Jun 16, 2026, 5:07 PM UTC

adam_arthur wrote: post #135

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Gemma 4 is particularly good at pipeline/automation tasks. It outperforms all the Qwen models (even 100B+) for rule following/automation style tasks in my experience. Its image interpretation is also very good, and out-benchmarks Opus. Qwen seems to ignore instructions and consistently outputs incorrect formats (when token generation format is not explicitly constrained) But yes, on the DGX Spark Gemma 31B Q4 with MT…

Post by CamperBob2 » Tue, Jun 16, 2026, 5:08 PM UTC

sathackr wrote: post #30

rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…

Post by cogman10 » Tue, Jun 16, 2026, 5:10 PM UTC

regularfry wrote: post #186

cogman10 wrote: post #164

Earlier quoted context omitted.

I believe there's a level of diminishing returns. Sure, SOTA will probably always benchmark better than local models. But do we need it? That's the question that the likes of OpenAI and Anthropic should be worried about.

The difference won't be in the individual tasks. It'll be in the scale of job they can take on and how you interact with the model. Think of pairing with a junior vs replacing a full delivery team, that's the sort of difference we'll be looking at. We'll be able to get closer to the latter by being more clever with harnesses, I reckon, but the frontier labs will run ahead because for any given harness trick they can…