Earlier quoted context omitted.
Does not apply to oss models
Does it not apply to hosting and running them for money? How would it not?
It focuses on training and data provenance, rather than serving.
Running local models is good now
541–550 of 651 posts
Re: Running local models is good now
#542I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…
This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…
My system is quite similar to your, my GPU is a 6950 XT and CPU a Ryzen 5 2600x, same amount of RAM, and I feel your pain. It sounds very similar to my experience from a few months ago. When it comes to tool calling, there are multiple possible issues; some models have borked templates bundled with the model file, some models are not trained on tool calling, some agent harnesses doesn't support the tool call output from the model very well, some quantizations ruin the models' abilities to call tools.
My suggestions if you want to further experiment with local models are to use llama.cpp instead of ollama [1], learn a little about the parameters that tune how much VRAM is used [2], look online for jinja template fixes for the model you're testing [3], and choose a model that was designed to do the task you want to achieve, with as high quantization as you can fit. The maximum model size you can run is VRAM + RAM, although you want as little of the model to be in system RAM as possible.
I'm running North Mini Code IQ3_XXS with some tuned parameters to fit my current tasks, and while it is not perfect for everything, it has not failed any tool calls I've asked it to make, or that it figured it should make on its own.
[1]: https://sleepingrobots.com/dreams/stop-using-ollama/
[2]: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
[3]: https://gist.github.com/jscott3201/e4b155885cc68c038d6ac8909...
Re: Running local models is good now
#543Earlier quoted context omitted.
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
Hetzner workforce can barely run a mature technology called s3 and you think they will be able to deploy openmodels?
What mature implementations of S3 are there? MinIO that rugpulled the community, Garage that doesn’t even have proper setup scripts in their Docker containers and expect you to do the init manually, or Zenko cloud server that more or less got abandoned? I think there’s also SeaweedFS which might do better but I’m surprised at how shitty everything seems in this space - surely people aren’t being crazy and either storing their files on the FS directly to expose access to them through their app (hello directory traversal attacks) or storing them in relational DBs (hello wasted bandwidth and bloated backups).
The odd jank extends further, like Sonatype Nexus and some other software hardcodes AWS regions to choose from when configuring the storage even though your self-hosted implementation doesn’t have anything to do with AWS so you just have to come up with fake regions. If the cloud vendors each have to reimplement it because there is nothing as quality as PostgreSQL is for DBs, but for S3, then I’m hardly surprised at the state of things.
Re: Running local models is good now
#544[flagged]
Re: Running local models is good now
#545I don't run local models, my devices are 5-6 years old and not powerful enough. It's a bit counter intuitive and different to what a lot of engineers are doing but I don't have a mac mini, I don't have a powerful laptop, a lot of my dev work has always been cloud based, on github, on a VM, I'm mostly using SSH from my laptop and now Claude Code on my phone (exe.dev is hands down the best experience I had on this front when the agent is literally on the VM).
In an ideal world, yea you can run local models, but I need a powerful always on device for that, or the latest gear, and it will never be as fast as what I can use from google, anthropic, or through an API call. I really wish it was different but I have to shell out a ton of money for that, and I guess it's usecase specific right. Maybe if my phone was super powerful and could run models that would be great, but then I have this issue with cloud sync and using things anywhere else. There will be a world in which local models and self deployed models make sense, this is going to be a core experience, but I personally can't run them.
Re: Running local models is good now
#546[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…
[flagged]
Re: Running local models is good now
#547Do you all use local models only for coding? What about using them as decision assistant? For instance, I work in science and sometimes I have many scattered ideas that I'd like to feed into an LLM so I can refine them and extract a meaningful research question. Are local LLM suited for that task?
After predictably failing at generating a sewing pattern, Gemini gave me yesterday this excuse:
> Because AI generates pixels based on visual patterns rather than mathematical geometry, it creates the illusion of a sewing pattern without any of the functional blueprints required to actually drape and construct a real garment.
If you want the illusion of a meaningful research question then sure, local models will give you that.
Re: Running local models is good now
#548After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…
Re being away from the HW: with Tailscale and llama-server it's now super easy to just run an inference server at home and use it from wherever you are.
Re: Running local models is good now
#549Earlier quoted context omitted.
A million times this. There is “private” as a corporate-legality licensing perspective. There is “private” as a human concept. The two are seemingly opposite, yet as all the money is focused on the former there’s no airtime left for the latter.
Then I'm interested if there are any facts as to what ZDR actually means?
It can still mean Zero Data Retention - i just comes down to whether you trust the company to actually do what they promise.
The fact that they've trained models on data that wasn't theirs does not make me trust them a lot when they make this claim.
Re: Running local models is good now
#550Earlier quoted context omitted.
This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…
My system is quite similar to your, my GPU is a 6950 XT and CPU a Ryzen 5 2600x, same amount of RAM, and I feel your pain. It sounds very similar to my experience from a few months ago. When it comes to tool calling, there are multiple possible issues; some models have borked templates bundled with the model file, some models are not trained on tool calling, some agent harnesses doesn't support the tool call output f…
"My suggestions if you want to further experiment with local models are to use llama.cpp instead of ollama"
Or at least LM Studio if you want to play around with a lot of different models. Im currently using it with my 7800xt and Vulcan as i found it left my OS more stable ROCm does. I had a few system crashes with ROCm and running out of VRAM for the OS.