Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

491–500 of 651 posts

Re: Running local models is good now

#491

Post by DiabloD3 »

orf wrote: post #102
fg137 wrote: post #91

> I have a 2022 M2 Mac with 64 GB RAM I closed the article after that. The author has no idea what a privilege it is to have a machine like that for personal use , and how 99% of the population are not going to afford a setup like that. Just some back-of-the-envelope maths will tell you that a $20/month Claude subscription makes much more sense financially.

99% of the population don’t code using models, local or remote. So that’s a useless metric. What % of developers could afford an older MacBook model, second hand? Far, far more than 1%.

But why would developers _step down_ to a Mac?

Re: Running local models is good now

#492

Post by DiabloD3 »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

[dead]

Re: Running local models is good now

#494

Post by EagnaIonat »

aftbit wrote: post #354
EagnaIonat wrote: post #262

Earlier quoted context omitted.

Depends what you need the model to do. The recent granite4.1:3b just takes 2GB of memory and is fast. Results are pretty good and support tool calling. Barely a squeak out of the Mac laptop. Even faster with the MLX builds. Then when I need more heavy lifting I fire up a larger model. IMHO the issue isn't the models. I've had OpenClaw give the same results as Claude using open models locally. Slower but does the job.…

Yeah it 100% depends what you want the model to do. Some tasks, like extraction, summarization, or simple tool calling (e.g. "turn on my desk lamp") are very doable with tiny models. Others, like coding or more advanced agentic workflows can demand much more powerful models. I was thinking from the lens of coding or running _big_ data extraction pipelines (think ~8 billion pages).

> thers, like coding or more advanced agentic workflows can demand much more powerful models.

You can do coding and agentic fine. For coding I use qwen3.6:35b-mlx and agentic granite4.1:3b works fine.

These are the models I use.

- granite4.1:3b

- granite4.1:30b

- gpt-oss:20b

- gpt-oss:120b (less so now)

- mistral-small3.2

- qwen3.6:35b-mlx

There will always be use cases that don't sit on your laptop, but most of what can be done can be done locally, it just requires a good framework to sit on it.

Re: Running local models is good now

#495

Post by boguscoder »

nateb2022 wrote: post #177
greenavocado wrote: post #143

Earlier quoted context omitted.

I have a 5070 12 GB laptop GPU and can hit 72 tokens per second in the first couple thousand tokens before dropping to mid-high 50s after about 15k context. This setup is extremely optimized down to the last flag. Changing any param above the temp flag craters performance. I don't have enough system RAM to properly handle the large context windows so I don't use local models. # 1,257 tokens 17s 72.18 t/s $env:CUDA_DE…

I get over 100 tok/s sustained on my M4 Max and M5 Max, in MacBook Pro's. LM Studio + MLX.

Same experience on M4 Max .. but quality of qwen still leaves so much to be desired after getting used to virtually unlimited tokens at work. Many people on this (and similar) thread seem to believe local models would inevitably improve, and I want to believe this too, but I don’t see this ever happening without growing in size

Re: Running local models is good now

#496

Post by abalashov »

FuriouslyAdrift wrote: post #233
c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

Kimi 2.6 or 2.8 is what we are playing with locally. They need 512GB to 1TB to run with full capabilities so that's not exactly "desktop" Our GPU computer server cost $110k.

But boy, it must be glorious. I use these through OpenRouter and rarely bother with Claude anymore.

Re: Running local models is good now

#500

Post by larodi »

milesvp wrote: post #463
rapind wrote: post #226

Earlier quoted context omitted.

> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…

If you think your data isn’t being hoovered up I’d like to point out that every model is possible due to federal crimes committed to obtain the information they were trained on. Regardless of how much you are paying, your data is worth another petty civil infraction.

The curiosity is that these companies somehow got around crimes and are above law (1) and these crimes mean something in a limited jurisdiction, like copyright laws of USA/Canada are not world’s (2). So it’s all cyberpunk at this point.
Post reply on HN