Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

521–530 of 651 posts

Re: Running local models is good now

#521

Post by frobisher »

sathackr wrote: post #30
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…

I suppose cloud won because: - nobody wants to deal with the networking stack on the internet - you want servers alive all the time - it's businesses running their software on servers to serve to customers

Do these apply to AI?

Re: Running local models is good now

#524

Post by andwhatisthis »

ios-contractor wrote: post #508

I subscribe to this guy on youtube for local model stuff if anyone is interested https://www.youtube.com/@AZisk . I'm not affiliated and I'm not even a paying subscriber. But I like all stuff local.

I clicked and immediately subscribed, but then checked out his latest videos and was so put off by the stereotypical clickbait stuff (stupid faces on thumbnails, "I tried (...) and then THIS happened" etc) that I unsubscribed. I understand that it must be what one needs to do to maximize views and brown nose the recommendation algorithm but I just find it incredibly off putting

Re: Running local models is good now

#525

Post by phunterlau »

schmuhblaster wrote: post #451

I’ve been playing around with qwen3.6-35b-a3b and managed to boost it significantly by leveraging my own custom harness [0]. It is quite astonishing to see how far local models have progressed, and I think that if you enjoy tinkering a bit, you can save a good bit of money (if you happen to have the hardware lying around anyways). Overall it’s still hard to beat the the cost/convenience combination of a cloud based m…

Cool, so the determinstic harness can boost the agent pretty much!

Re: Running local models is good now

#526

Post by iLoveOncall »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

[dead]

Re: Running local models is good now

#527

Post by rob74 »

rapind wrote: post #263
darkmarmot wrote: post #251

Earlier quoted context omitted.

Hard to guarantee it's private if you don't keep it local... I don't have a lot of trust for companies in this space.

Yes, but I think that'll change eventually. If you trust hosting your code with a specific cloud provider then you'll probably also trust them for code assist. At least that's my theory. There'll probably need to be a threat of massive litigation should they fail to comply with such a policy.

My company has all the code in a private GitLab instance (almost everything else is on AWS, but not GitLab), but they still use Cursor, so our internal code gets sent to whatever AI company the model I select in the dropdown belongs to. Scary if you think about it: if you use Cursor, you don't have to trust only one specific AI company, you have to trust all of them...

Re: Running local models is good now

#528

Post by xlii »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

> I use a lot of local models and they're still pretty painful to run locally.

This really depends on how and what you're using. e.g. I can't suffer through slowness of inference on Macbook but I have gaming rig with quite powerful GPU and I squeeze ~130 t/s on Gemma or ~70t/s on Qwen.

Tuning is not optional as well. Qwen on temperatures > 0.5 is unusable for coding and I found sweet spot around 0.32 for coding. Speculative decoding on Gemma4 26B is a 30t/s difference between non-speculative.

The worst thing with local models is that I can't just give you a recipe, because what's the best params depends on your use case.

In the nutshell I'd compare local models to running game rig on Windows vs Linux. Linux works great if not better than Windows gaming, but you need to embrace some tweaking in order to get there. Is it there? It's not SOTA, that's for sure, but it's working reasonably well.

Re: Running local models is good now

#529

Post by monegator »

cheschire wrote: post #461
pornel wrote: post #344

[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…

Haves and have nots. We aren’t wealthy enough to have the hardware that would make this good. The people who have the money to buy a spare maxed out Mac mini just don’t get it. I see lots of folks with RTX 6000’s in threads like these. Or any RTX card that ends in “90”. Cloud AI is what allows the proles to participate in the broader AI conversation, but not these AI conversations.

But cloud is what will enslave them to the corporation's will.

Google (of all companies!) demonstrated you can get useful stuff with reasonable performance with model running local on their smartphones.

Depending on your expecations you can get the local models running on a recent enough laptop, you just need 16GB of ram to be comfortable. It certainly exceeded my expectations (but i don't use the LLM to write code, only to do the real boring stuff: docs.)

Re: Running local models is good now

#530

Post by sieste »

The "middle powers" (cf Carney) should invest in local models, rather than relying on US and China allowing them to rent their AI models. It takes a single executive order to cut the rest of the world off of American AI tools. "I'm happy to pay whatever to rent frontier models from hyperscalers" makes sense if you're citizen of a superpower, but it's risky, naive, bordering on irresponsible to adopt this mindset otherwise, especially when your business or career depend on the tool.
Post reply on HN