51–60 of 91 posts
The idea that you need a new machine is pretty ridiculous. I bought a used HP Omen with a 3090 last month for $2k. 57t/s with Qwen 3.8.
Earlier quoted context omitted.
You can't run recent openAI/Anthropic models locally anyway, so wouldn't a better comparison be a different provider running Qwen or similar model? As then you can also compare against the exact model you'd have locally and any different data privacy of that particular provider ?
GP's point is about "sending tokens to someone else's computer" versus "keeping the tokens locally". I think model capabilities are secondary. In May of this year, I was running qwen3.6:35b-a3b on my MacBook (bought in 2024). Obviously not as fast as, say, running a model on Cerebras, but a year ago it wasn't really feasible to have a local model running on my 2024 laptop with vision support. (Concretely, I was passi…
Not a fair comparison really. If you can run a model locally then you can somewhat train out the guardrails, censorship, and brand-safety. That has value a subscription does not. Idk about the quality of this setup but just pasting it here as an example. https://explainx.ai/blog/heretic-llm-abliteration-guide-2026
> If you can run a model locally then you can somewhat train out the guardrails, censorship, and brand-safety. When does the average person actually need to do that?
Case in point, last week I was poking Opus 5 into writing me some RPi-pico firmware for driving a small e-paper screen. Font was built in right into C code as hex constants. Space being tight, I asked if there is some clever compression that could be applied. Claude thought for good 10 minutes, then guardrail kicked in telling me that was "cyber", and refused to continue.
Earlier quoted context omitted.
> If you can run a model locally then you can somewhat train out the guardrails, censorship, and brand-safety. When does the average person actually need to do that?
We've already seen frontier models refuse to answer almost any question that touches on computer security and be very likely to kick out biology and chemistry questions even if they aren't all that close to breeding dangerous viruses or making explosives. I expect this is only going to get worse. "Censorship" isn't just going to be about who you vote for and which political party the model will say nice things about…
I sell those to business who need local air gapped requirments and I make a lot more money!
I can run the alliterated models where none of the service prvoider even dare to provide.
THose benefits outweights a few K.
And show me an api provider that allows me to run 10x agents concurrently for 5 days straights .
Earlier quoted context omitted.
> If you can run a model locally then you can somewhat train out the guardrails, censorship, and brand-safety. When does the average person actually need to do that?
I just got some kind of cyber alert from Claude and was forced back down to Opus while I was trying to connect to a battery I own via bluetooth. So I can certainly understand why someone would want the guardrails gone.
It pays off instantly , because OpenAI/Anthropic can no longer see what I'm doing and that's worth a lot of money to me. If I am offloading some of my thought processes to a machine, I want to own that machine. And if I finetune the model, I can gain access to parts of thought space that are cordoned off by OpenAI/Anthropic/Alibaba/whomever due to their "alignment" efforts (i.e. alignment to the AI company rather tha…
You can't run recent openAI/Anthropic models locally anyway, so wouldn't a better comparison be a different provider running Qwen or similar model? As then you can also compare against the exact model you'd have locally and any different data privacy of that particular provider ?
You are missing out a lot.
Try that with Hermes or Opencode or Deekseek Harness , even Qwen 3.8 27b works really well for that kind of that.
I just ask it to install windows as a vm on my linux and install vs Community 2019 on it , and then build a legacy vb 2019 project on it. and sleep
When i wake up :
It installs Qemu , setup a vm , inside vm download and install windows 10 on its own , clicking next next next as needed , typing in things , writing powershell , python scripts , that run automatically after install by baking into CD that includes ssh server , reboot , it logins into ssh , trigger pythons script that continue installation of vs 2019 community , which includes a driver that click the installation steps , installs nuget , install all depedencies and then build the project into exe after i woke up.
That is with 100% pure local AI .
Local LLMs are not really about saving money, they're about autonomy. Choose the exact model you want, fine-tune it if you want, and no one can take it away from you.
Earlier quoted context omitted.
Technically true, but the delay between local and closed frontier is only a few months. And individual sovereignty / digital bodily integrity is almost priceless.
Local frontier costs a half million dollars to run locally in anything higher than basically ternary.
Earlier quoted context omitted.
GP's point is about "sending tokens to someone else's computer" versus "keeping the tokens locally". I think model capabilities are secondary. In May of this year, I was running qwen3.6:35b-a3b on my MacBook (bought in 2024). Obviously not as fast as, say, running a model on Cerebras, but a year ago it wasn't really feasible to have a local model running on my 2024 laptop with vision support. (Concretely, I was passi…
Wouldn't the better comparison still be against an AI provider with better privacy controls, especially if that's what someone cares about (even if they don't care about whether they're comparing a 35 billion param model vs a x trillion param model)?
What you can possibly audit is probably data sovereignty. For instance, I would not be surprised if Mistral's customers demand concrete evidence that their data is held within the European Union. But that is a distinct issue from training on input tokens.