I think no AI needs to be the norm. Even if we have enough RAM to run it locally, the dependency stack we have on hardware, training and geopolitics is too much of a risk to take on. If something breaks, like supply chain, or the model is found to have particular bias or exploits baked in, we're fucked.
Local AI needs to be the norm
551–560 of 804 posts
Re: Local AI needs to be the norm
#552Earlier quoted context omitted.
Tbf I've always hated that about HN formatting as it's not very clear at all that that's how it works. If there's a newline in my comment, why not retain it? Whyyyyy?!
Because of the 6 people who write HN comments in Vim with hard wrapping turned on.
That isn't really a problem because the comment should just be posted hard-
wrapped. It looks better that way.
Re: Local AI needs to be the norm
#553Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.
Local models embody the hacker spirit, constant Claude glazing is spiritually incompatible with tinkering. Don't upload your spirit to the cloud.
But if I run the model locally I have to pay for it, whereas with Claude I can, oh wait I just hit my 5 hour free limit with 2 messages.
Re: Local AI needs to be the norm
#554Earlier quoted context omitted.
> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…
> they don't have a used by date For quite a lot of use cases, the current systems arguably do get worse over time if not continually updated. The knowledge cutoff date will start to hurt more and more as the weights age in a hypothetical scenario where you are stuck with them forever. Coding, one of the most popular usescases today, would not be great if it say only understood java to a version from years ago etc. h…
I genuinely don't understand how can this possibly be a problem long term.
It feels very obvious that the solution is to have a smaller model that can be trained exclusively on Java information to augment the older model. If the architecture doesn't support it currently, then that's what the architecture will look like in the future.
Otherwise you'd be arguing that, to serve users who want to an up-to-date LLM on topic X, you have to train the model on the entire ABC all over again.
It's simply ludicrous to have a coding LLM that needs to be retrained on the latest published poems and pastry recipes to generate Java.
Re: Local AI needs to be the norm
#555Running LLMs locally is one way to realize the level of hardware and infrastructure that frontier AI companies are running. Makes me wonder about future strategies.
As one commenter mentioned, 2x Mac Studio M3 Max with 512GB can run frontier models and it costs $30k (with RDMA). Apply an efficiency ratio for being in a datacenter, and you understand why OpenAI and the likes spend north of $10k _per customer_ of CAPEX.
Add to that the electricity costs and you've got a very shaky business model. I for one would like to thank the VC for subsidizing my tokens.
With that said, the VCs are not crazy and probably factored in an annual cost decrease of computing power. But how do you make sure that we won't run local LLMs when the HW becomes affordable -- if ever ?
The answer has always been the same in our industry: vendor lock-in. They are getting the users now at a loss, hoping for future captive revenues.
So, be careful when your code maintenance requires the full context that yielded that code, and that this context is in [Claude Code|Codex|Cursor].
Re: Local AI needs to be the norm
#556Earlier quoted context omitted.
> paying $100/month There will not ever be a monthly subscription for LLM tokens. The economics isn't there. Local tokens will always be cheaper.
What's the basis for saying local tokens will always be cheaper? As others have outlined, LLMs serving one user at a time are pretty expensive, but concurrent users become much more cost-effective (assuming there's enough RAM for the contexts). If "local" to you means ~10 hours daily use by a team of employees, the company still has to balance against cloud services that can amortize non-recurring costs over 24 hours…
Why would a team of employees not be able to run AI workloads 24/7? Not all workloads are time sensitive.
Re: Local AI needs to be the norm
#557Earlier quoted context omitted.
Tbf I've always hated that about HN formatting as it's not very clear at all that that's how it works. If there's a newline in my comment, why not retain it? Whyyyyy?!
Because of the 6 people who write HN comments in Vim with hard wrapping turned on.
as a vim user I feel unnecrserrily called out
Re: Local AI needs to be the norm
#558Earlier quoted context omitted.
> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…
I'm sorry to spoil it for you, but Perl script was able to do all of that like ... 10 years ago? The out-of-the-box Shotwell manages photos quite well without any intelligence. The problem, as people mentioned above, is SOTA models cognitive and tooling abilities. Also, have you noticed as top-end Mac Studios got downgraded recently? They don't want you to have access to frontier models. And you will not have it. See…
> The out-of-the-box Shotwell manages photos quite well without any intelligence.
This piqued my interest on how it does it and after briefly checking the project it seems it only has two features for automatic photo categorization. 1) it can group photos by date and 2) It has face detection and recognition that uses trained weights (so ML "intelligence").
Re: Local AI needs to be the norm
#559Earlier quoted context omitted.
> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…
Can you share how you use it to categorize trip photos!
I'm also interested on how to do this
Re: Local AI needs to be the norm
#560Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.
Local models embody the hacker spirit, constant Claude glazing is spiritually incompatible with tinkering. Don't upload your spirit to the cloud.
They may well do but in practice if you want to embody the hacker spirit, the best thing is to hack rather than trying to get some clearly inadequate local LLM to do it.