Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

451–460 of 651 posts

Re: Running local models is good now

#451

Post by schmuhblaster »

I’ve been playing around with qwen3.6-35b-a3b and managed to boost it significantly by leveraging my own custom harness [0].

It is quite astonishing to see how far local models have progressed, and I think that if you enjoy tinkering a bit, you can save a good bit of money (if you happen to have the hardware lying around anyways). Overall it’s still hard to beat the the cost/convenience combination of a cloud based model provider though.

[0] https://deepclause.substack.com/p/how-to-make-small-models-p...

Re: Running local models is good now

#452

Post by fc417fc802 »

naikrovek wrote: post #444
rapind wrote: post #263

Earlier quoted context omitted.

Yes, but I think that'll change eventually. If you trust hosting your code with a specific cloud provider then you'll probably also trust them for code assist. At least that's my theory. There'll probably need to be a threat of massive litigation should they fail to comply with such a policy.

> Yes, but I think that'll change eventually. Maybe people will trust companies, but those companies will rarely deserve that trust. Anyone that pays attention sees breach announcements almost every day. Security is never a concern for these companies until it embarrasses them. Then, as soon as the negative attention fades, security again becomes the second to last priority. Do not trust companies with any data that…

If your contract says there's no data retention and then a bunch of your retained data gets leaked in a breach presumably you have grounds for a lawsuit.

Re: Running local models is good now

#453

Post by RevEng »

richardwhiuk wrote: post #220
preommr wrote: post #82

Earlier quoted context omitted.

> The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. AI is different. Cloud computing genuinely is cheaper on average. It's better than paying for cisco servers, and at scale, it's cheaper than managed platforms (ala Heroku), and it's a coin toss for when you're in the middle ground and constantly approaching the point of rebuilding poor-man versions of…

There's no economic reason why running a model locally should be better than using a cloud hosted version.

You pay a 3x markup to rent a server through AWS than managing your own. You pay for convenience. At shall annals that's fine, but for large companies with their own datacenters, you generally do things in house.

Re: Running local models is good now

#454

Post by fc417fc802 »

boppo1 wrote: post #437
saghm wrote: post #136

Earlier quoted context omitted.

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…

I have almost your system specs, how do they work for non-coding stuff like chat/knowledge/discussion? I've been using models to talk through social stuff I'm anxious about but dont want to annoy my friends with and it's been amazing, but I don't want to share that info with google/openai/anthropic anymore. I shouldn't have in the first place, but I couldn't help it, the exercise was too interesting.

You can test the open models for yourself using the various router services. Those also make it easy to use providers other than the major players.

Re: Running local models is good now

#456

Post by dakolli »

It doesn't make sense, if your small local model is 75% as effective as a frontier model and frontier models are still what.. 50% effective maybe slightly more, with tons of downsides.. Why would I spend 5k on hardware to run these mediocre models. I don't really see the point in the frontier model either.

Re: Running local models is good now

#457

Post by rapidfl »

sathackr wrote: post #30
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…

There is efficiency in the cloud model for models. So maybe there is a scope for Apple or an "Apple for AI" in the AI compute game - mainly from the perspective of privacy etc.

And once the servers are in space, everything is fully out there.

Re: Running local models is good now

#458

Post by linuxhansl »

I soooo wish that to be true. Alas, in my experience it is not... Yet.

What is true is that it gets easier and faster to run local models. With QAT (quantization aware training), turboquant (or similar) K/V compression; what used to be impossible to run is now fairly easy.

I can run gemma4:26b-a4b-qat on my laptop with 20-30 tokens/s with a 256k context window. That was unthinkable just 6 months ago.

So the local models are "OK" for small'ish projects.

But it does not at all(!) compare to the frontier models. For a large project Claude's Opus 4.6+ just work, whereas local gemma tangles itself up, makes weird mistakes, and just can't handle it (for those cases it is faster if I do it myself).

If the trends continues, with 1.58bit QAT models, even better K/V compression, faster multi-token prediction et al, maybe soon it will be comparable.

Re: Running local models is good now

#459

Post by fragmede »

rsanek wrote: post #429
angry_octet wrote: post #394

Programmers are used to paying nothing for tools. A basic laptop (SSD, multi core, 16GB of RAM) is hugely powerful if you are building in C/C++/Rust, even python. But all of a sudden it's no good, and we're back to using someone else's computer, hiring our tools every day. Worse, we get a different model every day, and maybe we aren't allowed to borrow the good tools some days because some mafioso are shaking down th…

Who's paying the $50k? I don't see how it makes sense to pay that much for a home-grown setup when I could pay <$5k/year total for both of the two best frontier models at effectively unlimited usage.

> best frontier models at effectively unlimited usage.

It would've been easy to spend $5k on Fable in the short week it was available. If that's the direction things are going (we can assume GPT-6 to be if similar class) $5k's not going to get you "best frontier models at effectively unlimited usage".

Re: Running local models is good now

#460

Post by linuxhansl »

hypfer wrote: post #15

After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…

How qwen3.6:27b compare to qwen3.6:35b-a3b (MoE) in your experience (if you tried). I find the dense models are way too slow on my H/W.
Post reply on HN