I very much enjoy scrolling r/homelabs ever so often, so many cool local rigs there
Running local models is good now
591–600 of 651 posts
Re: Running local models is good now
#592I love that local LLMs are being discussed more often on HN recently. But for the post, I find it strange that the author claimed they were working with local models from day 1, but wrote a post that still links to Qwen2.5 and Qwen3 in mid June 2026.
Why shouldn't the author mention models that people might not have to buy a new computer to use?
One don’t have to buy a new computer to run Qwen3.6 or Qwen3.5 (35B A3B), given that they can already run Qwen3 30B A3B.
In fact with a 64GB mac, you can run pretty much all of the latest Qwen models.
Also, anyone who has been following local LLM are well aware that the quality and performance has become way way better since Qwen3.5
Re: Running local models is good now
#593[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…
Those who have never known anything better are okay with much less. For example, anyone who used Fable when it came out are saying that it is very difficult to go back to lesser models now. Even our strongest aren't good enough in comparison.
I used fable, and directly compared it against sonnet 4.8 and gpt 5.5 on various tasks. It was generally better, but still not perfect. Going back to sonnet/gpt has been perfectly fine.
Re: Running local models is good now
#594Earlier quoted context omitted.
> Yes, but I think that'll change eventually. Maybe people will trust companies, but those companies will rarely deserve that trust. Anyone that pays attention sees breach announcements almost every day. Security is never a concern for these companies until it embarrasses them. Then, as soon as the negative attention fades, security again becomes the second to last priority. Do not trust companies with any data that…
If your contract says there's no data retention and then a bunch of your retained data gets leaked in a breach presumably you have grounds for a lawsuit.
That may be true, but it is likely that the contract limits you to arbitration, forbids class action and limits the penalty dramatically.
Re: Running local models is good now
#595> For my local setup, I’m currently [..] and LM Studio as the inference server, although it would likely be faster if I just used llama.cpp directly
Is there any truth to this claim? LM Studio uses llama.cpp to run the models. I guess the overhead of LM Studio should be minimal.
After all LM Studio is a really easy way to host models, are there really major drawbacks?
Re: Running local models is good now
#596Earlier quoted context omitted.
It can still mean Zero Data Retention - i just comes down to whether you trust the company to actually do what they promise. The fact that they've trained models on data that wasn't theirs does not make me trust them a lot when they make this claim.
When discussing this, may I ask (I know you are probably bored of the actual arguments), what does "trained models on data that wasn't theirs" actually mean in practice? Again, I know these arguments have been done to death, but every human who reads source code that wasn't written by them, or views art that wasn't created by them, and practices against this art, is training their brain on data "that wasn't theirs".…
A product is not a human. They are selling a product based off copy-righted material without the rights to it. It's a pretty easy line to draw, honestly.
Re: Running local models is good now
#597Earlier quoted context omitted.
Yeah it 100% depends what you want the model to do. Some tasks, like extraction, summarization, or simple tool calling (e.g. "turn on my desk lamp") are very doable with tiny models. Others, like coding or more advanced agentic workflows can demand much more powerful models. I was thinking from the lens of coding or running _big_ data extraction pipelines (think ~8 billion pages).
> thers, like coding or more advanced agentic workflows can demand much more powerful models. You can do coding and agentic fine. For coding I use qwen3.6:35b-mlx and agentic granite4.1:3b works fine. These are the models I use. - granite4.1:3b - granite4.1:30b - gpt-oss:20b - gpt-oss:120b (less so now) - mistral-small3.2 - qwen3.6:35b-mlx There will always be use cases that don't sit on your laptop, but most of what…
Why do you like gpt-oss-120b less now? What replaced it?
Re: Running local models is good now
#598> For my local setup, I’m currently [..] and LM Studio as the inference server, although it would likely be faster if I just used llama.cpp directly Is there any truth to this claim? LM Studio uses llama.cpp to run the models. I guess the overhead of LM Studio should be minimal. After all LM Studio is a really easy way to host models, are there really major drawbacks?
LM Studio has a lot less tuning options when you launch it. Also it is precompiled so you don't have the latest (and sometimes buggy) releases so you may have to wait a few days to try out a MTP or a new model. LM Studio is easier though.
Re: Running local models is good now
#599This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…
I know coding is the killer app thus far, but if businesses are seeing any kind of significant cost for other LLM usecases, seems like at least a decent consultancy opportunity to set up medium-sized businesses with in-house kit.
The other question is how the middle ground (hetzner etc) is shaping up, because obviously so many orgs won't want to run servers.
Re: Running local models is good now
#600Earlier quoted context omitted.
IMO running local models "well" still requires an expensive hardware investment. You really want 96GB of VRAM on a modern Blackwell arch to run these models with decent KV cache. Trying to run them on a unified memory Mac, an AI Max AMD processor, or a DGX Spark-alike is really just asking for trouble. Prefill kills perf. If you throw the right GPUs at the problem, they become much better - but still not quite in the…
For a fraction of the price of 96GB vram, I built a desktop based on a supermicro server mobo and EPYC 9 series CPU, with just under 400GB rdimm ram (approx $4500 all in but this was before the ram price hike). Works really well for serving larger local modals at a decent enough speed (I consider anything more than 10 tokens/second usable and value accuracy over speed).
that costs the same as 210 months (17.5) years of using codex 5.5 while not being near as good.