Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

711–720 of 804 posts

Re: Local AI needs to be the norm

#711

Post by dleslie »

CamperBob2 wrote: post #705
dleslie wrote: post #695

Earlier quoted context omitted.

European models are competitive, despite the concerns you raise. I don't need a model that can easily produce CSAM or reproduce copyrighted works verbatim in order to be productive.

EU models are not, in fact, competitive with US or Chinese models. And that's not how these things work. If you censor the model for one purpose, you will degrade it for others. We both know that the bureaucrats won't stop at either of those purposes. It's not in a censor's nature to walk away satisfied.

Mistral is competitive and useful.

And Anthropic is famous for putting guardrails on their models, and yet continue to lead.

We don't have to, and shouldn't have to, tolerate tools that easily produce csam or similarly undesirable output.

Re: Local AI needs to be the norm

#712

Post by platevoltage »

hparadiz wrote: post #699
platevoltage wrote: post #687

Earlier quoted context omitted.

Yes. I have tried this stuff. I really don't see how my use, or non-use of AI APIs changes this reality. Github Copilot announced it's going to per-token pricing in less than a month. I heard that on the internet by the way.

I'm watching people host models on things like LangSmith and OpenRouter for a fraction of the cost you are talking about. We have other people reporting their M4 Macs providing them with performance close to what they get with ChatGPT and Claude all locally with just a 24 GB M4 Mac. We already spend money on laptops. I can put in a ticket for an M4 Macbook Pro from IT right now.

Fantastic. I run local models too. This was specifically about APIs

Re: Local AI needs to be the norm

#713

Post by hparadiz »

platevoltage wrote: post #712
hparadiz wrote: post #699

Earlier quoted context omitted.

I'm watching people host models on things like LangSmith and OpenRouter for a fraction of the cost you are talking about. We have other people reporting their M4 Macs providing them with performance close to what they get with ChatGPT and Claude all locally with just a 24 GB M4 Mac. We already spend money on laptops. I can put in a ticket for an M4 Macbook Pro from IT right now.

Fantastic. I run local models too. This was specifically about APIs

Why do you think costs will go up when the competition is increasing while hardware prices per compute cycle go down? That's the part I don't get.

Re: Local AI needs to be the norm

#714

Post by pianopatrick »

zozbot234 wrote: post #709
pianopatrick wrote: post #384

Earlier quoted context omitted.

Currently I'm testing something like this just to see what happens. I have an old laptop with 4GB of RAM. I attached a USB drive with Gemma 4 31B model (which is 32.6 GB). Currently the laptop is running llama.cpp and trying to respond to a prompt by streaming the model from disk. The USB drive light is flickering, showing something is happening. It's been about 8 hours since I entered the prompt and I've gotten abou…

Wow, that's a true worst case scenario especially if the USB is just plain old USB 2.0 (max 480 Mbps) and/or if the drive is a spinning disk. How's the CPU doing, though? Is there any headroom given the USB bottleneck?

running top shows the process llama-cli taking 29% of CPU and 88% of memory, while process usb-storage is taking 9% of cpu and 0% of memory

Re: Local AI needs to be the norm

#715

Post by jononor »

oytis wrote: post #27
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

Hardware sales would be an excellent business model for open weights. Nvidia is already on it with their Nemotron models. Any new LLM/NPU hardware companies would want to so the same, if noone else does it for them (Chinese labs currently do).

Selling managed self-hosting solutions would be another. That is the business of that recent American company.

Selling fine-tuning services or similar adaptations is another. That is what Unsloth is going for, I believe.

Most likely any sound business strategy is going to be of "commoditize your compliments" type. There are many complementary products to open-weight - some probably not invented/discovered yet.

Re: Local AI needs to be the norm

#716

Post by greenavocado »

nicce wrote: post #462
bugglebeetle wrote: post #400

Earlier quoted context omitted.

Anthropic serves quantized versions of their models and you can run q8 locally.

I don't even use Sonnet anymore. Current feels worse than Claude 3.5 couple years ago. They have quantized that much? Switched to GPT 5.5, let's see how long it will stay good.

The problem with GPT is it doesn't have the same wholistic worldview of the problem space as Claude

Re: Local AI needs to be the norm

#717

Post by nicce »

greenavocado wrote: post #716
nicce wrote: post #462

Earlier quoted context omitted.

I don't even use Sonnet anymore. Current feels worse than Claude 3.5 couple years ago. They have quantized that much? Switched to GPT 5.5, let's see how long it will stay good.

The problem with GPT is it doesn't have the same wholistic worldview of the problem space as Claude

Hmm, what that means or how can you even measure it? At least for all my recent problems, GPT 5.5 has performed better.

Re: Local AI needs to be the norm

#718

Post by reissbaker »

otabdeveloper4 wrote: post #394
reissbaker wrote: post #336

Earlier quoted context omitted.

Users do not have an existing $80k of hardware, are not going to buy $80k of hardware for worse performance than paying $100/month, and models are continuing to grow in size while memory grows in price.

> paying $100/month There will not ever be a monthly subscription for LLM tokens. The economics isn't there. Local tokens will always be cheaper.

There already are many subscriptions for LLM tokens: OpenAI, Claude, Synthetic (shameless plug), Zai...

I'm not sure what you mean by "There will not ever be a monthly subscription for LLM tokens." That already exists!

Re: Local AI needs to be the norm

#719

Post by digitaltrees »

manmal wrote: post #475
digitaltrees wrote: post #269

Earlier quoted context omitted.

I built my own IDE and run my own model specifically to have private agentic coding. I can still access model APIs but I can be purely local if I want too. It’s amazing.

Curious, why did Zed with ACP not work for you?

Because I wanted the full ide on my iPhone so I can code while away from my laptop doing fun stuff with my kids. And I don’t like the Claude codex fire and forget approach.

The ide I built has a full terminal, file system, git integration and AI agent. It uses a private cloud Linux container that is persistent so I can install packages and do anything I want from any phone, computer or browser. It’s amazing that we live in a time where we can build custom software for ourselves just for fun. I will never have to worry about cursor or vs changing getting bought and moth balled like Atom (my favorite ide). I now own my tool and will forever.

Re: Local AI needs to be the norm

#720

Post by digitaltrees »

oslem wrote: post #569
Fokamul wrote: post #510

Earlier quoted context omitted.

I'm just guessing, but IDE which is using 3D acceleration just for stupid UI to run "smoothly", that is ridiculous. Who runs IDE with LLM agents accessing your local filesystem, on bare metal? Or am I alone to run everything LLM related on my VM just for development work. Then because of ZED genius decision, you need to share your GPU to VM, then some important features will not work, like snapshots. So you also need…

I would imagine running a local LLM for development isn’t as popular as using a hosted provider. I don’t personally host a local model, but I have shared GPUs and storage volumes with VMs and I didn’t see it as that much of a hassle. What kinds of problems are you running into? Doesn’t ghostty also use graphics acceleration? I was under the impression that rendering text is a relatively challenging graphics compute t…

I run local LLM on my MacBook together with frontier models for different tasks. I am in the process of setting up a 3 Mac studio system to serve AI to my team.
Post reply on HN