Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

371–380 of 651 posts

Re: Running local models is good now

#371

Post by delis-thumbs-7e »

Nobody asked, but I don’t think any of us should be using SoA models to code or to do pretty much anything at all. Instead we should develop open models to work on specific tasks and learn to code, write, draw etc. using fingers made of bones and brains made of flesh. Big corporations and research facilities can run them to generate code or math or whatever, with a bunch of specialists to check the output to be correct. Then again, even that might not be worth the costs (e.g. OpenAI’s 36B$ net loss last year), when the open models are so close and the whole AI scheme is running out of scams to pull.

There’s a lot of things we could use even quite small models for, which would not need an insane amount of computing power and memory, but too few of us is really researching them.

Re: Running local models is good now

#372

Post by otabdeveloper4 »

sathackr wrote: post #30
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…

> in the American business model

AI company valuations won't survive if they're only for the "American business model".

Re: Running local models is good now

#373

Post by otabdeveloper4 »

codethief wrote: post #287
TkTech wrote: post #119

Earlier quoted context omitted.

For many companies (country-dependent) that's not really why they use cloud services vs purchasing. It's tax shenanigans and business process overhead. OpEx vs CapEx, and a small (%) bump in the huge AWS bill no one will even notice or a $30k+ invoice for hardware that has to go through rigorous review and 3 departments. Same reason people pay for things through the AWS marketplace (like Vanta) instead of having to g…

Good point. Maybe there'll be companies that maintain your on-premise GPU cluster just like there are companies that service the coffee machine in your office?

> on-premise GPU cluster

Renting a GPU server from a cloud and hosting your own llama.cpp is the path of least resistance.

Re: Running local models is good now

#375

Post by smcleod »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Those dense models are pretty fast with MTP now. 40-70TK/s depending on your machine, that's faster than cloud models (although not as smart obviously).

Re: Running local models is good now

#376

Post by lmedinas »

Stagnant wrote: post #337
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

I've been using unsloth/gemma-4-31B-it-qat-GGUF daily for various small parsing and programming tasks using opencode and llama-server's front end. The past couple of weeks have made a big difference after google released the QAT variant and llama.cpp got support for MTP which means it is possible to now get 60-80 Tok/s with RTX 4090. The model fits in VRAM comfortably enough to keep it loaded even while browsing and…

the problem of that setup is that it will run out of context pretty quick. So for coding agent it will limit your workflow very fast.

Re: Running local models is good now

#377

Post by saghm »

ryukoposting wrote: post #253
saghm wrote: post #136

Earlier quoted context omitted.

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…

I found that, with the heavily quantized Qwen3 models I can cram onto my 3060 Ti, telling the model to use its tools in the system prompt made it a lot more likely to actually do it. YMMV of course, but give it a shot.

I did try this, and it was pretty hit-or-miss still. I even went as far as configuring context for Zed to inject into all conversations saying stuff like "If you need to read a file, call read_file NOW. Do not say you will read it", and it still didn't really make a huge difference.

Re: Running local models is good now

#378

Post by abalashov »

throwawayffffas wrote: post #218
abalashov wrote: post #201

Earlier quoted context omitted.

Not a ton. I'd say 64 GB minimal to play, 96-128 GB better.

Nah, you can run the 24b - 35b class with between 90k and 256k of context with about 40GB and they are pretty good. Especially the MOE variants fit neatly in 40GB.

Yeah, but then you need RAM for the rest of your OS and applications. I'd say 64 to be comfortable in the sense to which most HN users are accustomed.

Re: Running local models is good now

#379

Post by stemlord »

zozbot234 wrote: post #116
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Maybe we shouldn't be running these models on laptops with their thermally constrained form factor, and we shouldn't expect quick inference on a par with a large cloud-based platform either, at least not for near-SOTA model quality. It's still worth it to avoid becoming massively reliant on centralized services.

> It's still worth it to avoid becoming massively reliant on centralized services.

This isn't really good enough. Many of us need to get things done in a pinch and if our employers are already getting used to the idea of paying for enterprise subscriptions to cloud llm's then the local option needs to be good

Re: Running local models is good now

#380

Post by fg137 »

orf wrote: post #102
fg137 wrote: post #91

> I have a 2022 M2 Mac with 64 GB RAM I closed the article after that. The author has no idea what a privilege it is to have a machine like that for personal use , and how 99% of the population are not going to afford a setup like that. Just some back-of-the-envelope maths will tell you that a $20/month Claude subscription makes much more sense financially.

99% of the population don’t code using models, local or remote. So that’s a useless metric. What % of developers could afford an older MacBook model, second hand? Far, far more than 1%.

could or will?

I am pretty sure even among software engineers, much fewer than 1% are going to spend their money on that.

Most software engineers know how to spend their money responsibly.

Post reply on HN