[flagged]
Running local models is good now
431–440 of 651 posts
Re: Running local models is good now
#432This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…
The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…
> Everyone wants to shuck the chore and the responsibility.
Which gives all the power to the big techs. I'll never understand why the average company seems to have no problem with this.
Re: Running local models is good now
#433Earlier quoted context omitted.
outsource that headache along with the responsibility for it You know what gives me headaches? When I'm in the middle of a session and the model gets rug-pulled out from under me because somebody at the model provider didn't pay the Trump bill that month. Or when someone at the model provider decides that the curve-fitting algorithm in my graphics package looks a little too much like Skynet for comfort. Or when they…
What about inference suggests it naturally belongs in the cloud?
Inference basically looks like this (neglecting a whole bunch of stuff):
for t in tokens_in_context
for p in model_weights
do something with p*t
Obviously that's not how it really works in practice, but the point is, if you are only running one prompt at a time, each weight gets fetched, applied to the token being processed, and then never touched again until the next token is processed.
So when you submit a prompt to a model that's running a bunch of other peoples' contexts concurrently, it can reuse each weight multiple times before moving on to the next one:
for p in model_weights
for u in users
for t in u's context
do something with p*t
Worst case, in terms of energy efficiency, is a single user sitting around waiting for a single response. I don't feel like I'm explaining it well, but the core idea is that every time a weight is fetched from memory, you want to get as much work done as possible with it.
Re: Running local models is good now
#434Earlier quoted context omitted.
IMO running local models "well" still requires an expensive hardware investment. You really want 96GB of VRAM on a modern Blackwell arch to run these models with decent KV cache. Trying to run them on a unified memory Mac, an AI Max AMD processor, or a DGX Spark-alike is really just asking for trouble. Prefill kills perf. If you throw the right GPUs at the problem, they become much better - but still not quite in the…
> DGX Spark-alike is really just asking for trouble. Prefill kills perf. You're right that prefill kills perf, but shrug the GB10 has far more compute than it has memory bandwidth, so prefill isn't it's bottleneck.
I've seen the same, Sparks are great at non time-sensitive tasks. if you can set up a agentic loop that does not require human intervention, you can design around the memory bandwidth limitations
Re: Running local models is good now
#435Local models are good? Or are we saying that open source/open weights models are good? What I'm asking is, are they good because they are "local" or are they good because you can install and run them yourself, wherever you want? Same node, different node, different cluster, way out in the ether/cloud...
Re: Running local models is good now
#436Earlier quoted context omitted.
Maybe we shouldn't be running these models on laptops with their thermally constrained form factor, and we shouldn't expect quick inference on a par with a large cloud-based platform either, at least not for near-SOTA model quality. It's still worth it to avoid becoming massively reliant on centralized services.
> It's still worth it to avoid becoming massively reliant on centralized services. This isn't really good enough. Many of us need to get things done in a pinch and if our employers are already getting used to the idea of paying for enterprise subscriptions to cloud llm's then the local option needs to be good
For me I use only cloud for work. But I'd never trust any of my personal data to it.
Re: Running local models is good now
#437I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…
This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…
I have almost your system specs, how do they work for non-coding stuff like chat/knowledge/discussion? I've been using models to talk through social stuff I'm anxious about but dont want to annoy my friends with and it's been amazing, but I don't want to share that info with google/openai/anthropic anymore. I shouldn't have in the first place, but I couldn't help it, the exercise was too interesting.
Re: Running local models is good now
#438I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…
I had some local model FOMO, trialed for a few days, and tentatively arrived at the same conclusion. I can get a better ROI on the time I spent waiting and dealing with poor quality by just programming by hand myself instead.
Re: Running local models is good now
#439Earlier quoted context omitted.
What about inference suggests it naturally belongs in the cloud?
Inference basically looks like this (neglecting a whole bunch of stuff): for t in tokens_in_context for p in model_weights do something with p*t The expensive part is fetching each weight from memory, which is why VRAM/HBM is such a big deal. Conceptually, for a huge, dense (non-MoE) model, the inner loop might run a trillion times for every token generated. Obviously that's not how it really works in practice, but t…
That makes a lot of sense, thank you. I think a pirate cloud of local models could make sense, but that would be regulated into oblivion
Re: Running local models is good now
#440Earlier quoted context omitted.
The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…
> Everyone wants to shuck the chore and the responsibility. Which gives all the power to the big techs. I'll never understand why the average company seems to have no problem with this.
Did you build your own house using tools that you forged from iron-rich ore yourself? Did you grow your own wheat to make bread for your lunchtime sandwich today?
There's a reason most people pay other people to do these things for them.