[flagged]
Running local models is good now
531–540 of 651 posts
Re: Running local models is good now
#532Earlier quoted context omitted.
Nah, you can run the 24b - 35b class with between 90k and 256k of context with about 40GB and they are pretty good. Especially the MOE variants fit neatly in 40GB.
Yeah, but then you need RAM for the rest of your OS and applications. I'd say 64 to be comfortable in the sense to which most HN users are accustomed.
Sure sure, if you plan to run it on system ram instead of dedicated gpus then yeah you need an extra overhead to run your own stuff.
Re: Running local models is good now
#533The "middle powers" (cf Carney) should invest in local models, rather than relying on US and China allowing them to rent their AI models. It takes a single executive order to cut the rest of the world off of American AI tools. "I'm happy to pay whatever to rent frontier models from hyperscalers" makes sense if you're citizen of a superpower, but it's risky, naive, bordering on irresponsible to adopt this mindset othe…
Training DeepSeek was magnitudes cheaper than training the SOTA models it relied upon.
In theory, other countries should be able to replicate that effort and improve it.
Re: Running local models is good now
#534[flagged]
Re: Running local models is good now
#535I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…
When running on a GPU, dense models are shaping up to be the best way due to two things: - Maximum intelligence per VRAM (you dont have much) - Dense models can benefit from MTP to get an almost 2x speedup in decode (ie, a 27b dense model with mtp decodes at about the same speed as a MoE model with 14b active param model would). This is important because local llm rarely has parallel streams to batch together. When r…
The problem with batching local LLMs is not any inherent lack of multiple parallel sessions, but rather that local dGPUs lack the VRAM capacity to host KV-cache for several of those at once, whereas unified memory platforms broadly lack the compute headroom compared to memory bandwidth that would actually make batching useful.
(SSD streaming a larger-than-RAM model "solves" that latter issue very nicely because it radically slashes the equivalent to memory bandwidth so any saving on that becomes highly significant.)
Re: Running local models is good now
#536Earlier quoted context omitted.
If you think your data isn’t being hoovered up I’d like to point out that every model is possible due to federal crimes committed to obtain the information they were trained on. Regardless of how much you are paying, your data is worth another petty civil infraction.
A million times this. There is “private” as a corporate-legality licensing perspective. There is “private” as a human concept. The two are seemingly opposite, yet as all the money is focused on the former there’s no airtime left for the latter.
Then I'm interested if there are any facts as to what ZDR actually means?
Re: Running local models is good now
#537Do you all use local models only for coding? What about using them as decision assistant? For instance, I work in science and sometimes I have many scattered ideas that I'd like to feed into an LLM so I can refine them and extract a meaningful research question. Are local LLM suited for that task?
Re: Running local models is good now
#538The "middle powers" (cf Carney) should invest in local models, rather than relying on US and China allowing them to rent their AI models. It takes a single executive order to cut the rest of the world off of American AI tools. "I'm happy to pay whatever to rent frontier models from hyperscalers" makes sense if you're citizen of a superpower, but it's risky, naive, bordering on irresponsible to adopt this mindset othe…
Training DeepSeek was magnitudes cheaper than training the SOTA models it relied upon. In theory, other countries should be able to replicate that effort and improve it.
I'd love for European countries to embrace their priorities around people and environment and train a model on public domain data that can run efficiently on cheap hardware. Redefine benchmarks around these ideals and optimise towards them, rather than trying to enter a race that you can't possibly win and didn't want to run in the first place.
Re: Running local models is good now
#539Earlier quoted context omitted.
This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
Hetzner workforce can barely run a mature technology called s3 and you think they will be able to deploy openmodels?
Re: Running local models is good now
#540Earlier quoted context omitted.
This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
For me it's about the value of my time. I think that it's important that we have open models, but for getting real work done, my time is too valuable to waste it on subpar results or additional agent management when a max plan covers all the use I need. It's not worth quibbling over. If the cost / benefit ratio changes, I'll be looking harder at local set ups, but not at the moment.