What's the best practice right now for setting these up? We've been primarily using Nix/Flox to set up the models pretty quickly and at least with minimized amount of commands(biased Nix/Floxer) here and found it useful
Running local models is good now
611–620 of 651 posts
Re: Running local models is good now
#612Earlier quoted context omitted.
When discussing this, may I ask (I know you are probably bored of the actual arguments), what does "trained models on data that wasn't theirs" actually mean in practice? Again, I know these arguments have been done to death, but every human who reads source code that wasn't written by them, or views art that wasn't created by them, and practices against this art, is training their brain on data "that wasn't theirs".…
In general humans don't have perfect recall. Even people with what we might call a photographic memory don't have the ability to memorise millions of lines of code and output them with little effort. It hinges somewhat on the concept of how much you believe things are being learned and how much is just pattern matching and borrowing a solution from memory. Certainly in the early days of Copilot it was possible to get…
> how much is just pattern matching and borrowing a solution from memory.
It's easy to show that this is not the case. This is a well-known phenomenon in ML, known as generalization - specifically, compositional generalization. See e.g. https://research.google/blog/measuring-compositional-general... for a description - although note that that post is from 2020, and models have become much better at this since then.
People can "believe" what they want, but there's plenty of work that definitively falsifies beliefs about "borrowing a solution from memory".
Re: Running local models is good now
#613Earlier quoted context omitted.
A product is not a human. They are selling a product based off copy-righted material without the rights to it. It's a pretty easy line to draw, honestly.
Why is something being human or not relevant here?
Because the laws were written by humans for humans, and that's where they seem to be drawing the line
Re: Running local models is good now
#614[flagged]
Re: Running local models is good now
#615Earlier quoted context omitted.
I get over 100 tok/s sustained on my M4 Max and M5 Max, in MacBook Pro's. LM Studio + MLX.
With Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf? Also, funny lumping the M4 "and" the M5, I find them 15% to 45% different performance, depending. And for a good deal of work, an M3 Studio Ultra outpaces the M4 and ties the M5 on single work at a time, outpaces both doing multiple work at a time.
> With Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf?
Ahh no I'm using the MLX version, it's about 5-10% faster than GGUFs in my experience.
Re: Running local models is good now
#616Earlier quoted context omitted.
Hetzner workforce can barely run a mature technology called s3 and you think they will be able to deploy openmodels?
What mature implementations of S3 are there? MinIO that rugpulled the community, Garage that doesn’t even have proper setup scripts in their Docker containers and expect you to do the init manually, or Zenko cloud server that more or less got abandoned? I think there’s also SeaweedFS which might do better but I’m surprised at how shitty everything seems in this space - surely people aren’t being crazy and either stor…
I work on SeaweedFS. Let me know if see any bugs or just create a github issue.
Re: Running local models is good now
#617Programmers are used to paying nothing for tools. A basic laptop (SSD, multi core, 16GB of RAM) is hugely powerful if you are building in C/C++/Rust, even python. But all of a sudden it's no good, and we're back to using someone else's computer, hiring our tools every day. Worse, we get a different model every day, and maybe we aren't allowed to borrow the good tools some days because some mafioso are shaking down th…
Who's paying the $50k? I don't see how it makes sense to pay that much for a home-grown setup when I could pay <$5k/year total for both of the two best frontier models at effectively unlimited usage.
$5k? I work at a software company where one engineer has used that every month minimum.
Re: Running local models is good now
#618Earlier quoted context omitted.
A highly skilled carpenter may be able to 'get work done' by banging nails in with a heavy-bottomed cocktail glass, doesn't mean it's not painful to do so when it is continuously breaking and leaving shards of glass all over the workshop for you to find every day for the rest of your life until you clean up the mess you made using the wrong tool for the job.
More like, a highly-skilled carpenter can work miracles with a $6 hammer from the hardware store, while the pros on the commercial crew are using fancy compressed-air tools. The carpenter has to get up close and personal with the wood. He can't match the crew's throughput, but maybe that's not what he's trying to do.
I would say the hammer is no AI. Local models are the cheapest XKGYAGH electric nailer on Amazon that "works" but jams up all the time. The $20/mo cloud models are a nice DeWalt that gives an hour of jam-free operation but takes five hours to recharge. And if someone else is paying for it, one can use the heavy duty nail gun with a big generator and compressor on a trailer that can run all day.
Re: Running local models is good now
#619Earlier quoted context omitted.
For a fraction of the price of 96GB vram, I built a desktop based on a supermicro server mobo and EPYC 9 series CPU, with just under 400GB rdimm ram (approx $4500 all in but this was before the ram price hike). Works really well for serving larger local modals at a decent enough speed (I consider anything more than 10 tokens/second usable and value accuracy over speed).
that costs the same as 210 months (17.5) years of using codex 5.5 while not being near as good.
Yes, if your trust model allows you to use API providers or the big 3, you 100% should. They have better util than anything you self host, so they can be more efficient. On top of that, they're shoveling cash into the fire to try to capture marketshare, so they're offering inference for well below break-even costs.
The main reasons to use local models are:
1. Self-sovereignty & control
2. Data security
3. Offline availability
If none of those apply to you, then you should just use OpenAI or Anthropic.
Re: Running local models is good now
#620Earlier quoted context omitted.
> thers, like coding or more advanced agentic workflows can demand much more powerful models. You can do coding and agentic fine. For coding I use qwen3.6:35b-mlx and agentic granite4.1:3b works fine. These are the models I use. - granite4.1:3b - granite4.1:30b - gpt-oss:20b - gpt-oss:120b (less so now) - mistral-small3.2 - qwen3.6:35b-mlx There will always be use cases that don't sit on your laptop, but most of what…
Why do you like gpt-oss-120b less now? What replaced it?
It's very likely to hallucinate. I'm mostly using Gemma 4 31B now when I need something offline. It is a very strong model for its size.