Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

601–610 of 804 posts

Re: Local AI needs to be the norm

#601

Post by Hamuko »

aceazzameen wrote: post #566
Hamuko wrote: post #507

Earlier quoted context omitted.

> Also, have you noticed as top-end Mac Studios got downgraded recently? They don't want you to have access to frontier models. And you will not have it. Isn't that a function of RAM supply not being available now?

OpenAI did buy out the RAM supply to block competition. Arguably local models are one of its (smaller) competitors. Even if that weren't the case, every corp _needs_ you to be on a subscription.

They didn't really even buy the RAM. But there's pretty significant demand for RAM in general with data centers being planned left and right.

Re: Local AI needs to be the norm

#602

Post by screamingninja »

fennecfoxy wrote: post #514
satvikpendem wrote: post #333

Earlier quoted context omitted.

Please add double new lines as your formatting for the bullet point list makes it all one paragraph.

Tbf I've always hated that about HN formatting as it's not very clear at all that that's how it works. If there's a newline in my comment, why not retain it? Whyyyyy?!

We just need some Markdown love here. That'll solve everything!

Re: Local AI needs to be the norm

#603

Post by DrewADesign »

Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

And this is exactly what the LLM provider industry is fighting tooth-and-nail. It’s not only because it doesn’t directly contribute to their bottom line, it also directly opposes the idea that LLMs are going to replace entire workers rather than enhance the abilities of individual workers. What we’re headed towards would have been a killer product and probably still shifted a bunch of capital to the bazillionaires had these companies set more realistic goals rather than banking that they’d be the ones that won the war that “changed everythingTM”.

Re: Local AI needs to be the norm

#604

Post by robrenaud »

epicureanideal wrote: post #369
digitaltrees wrote: post #271

Earlier quoted context omitted.

I wonder if it really needs to be worse. I am playing with the idea of fine tuning a model on my exact stack and coding patterns. I suspect I could get better performance by training “taste” into a model rather than breadth.

I also wonder about JS only, Python only, etc models. Maybe the future is a selection of local, specific stack trained models?

There is some recent work on modularizing knowledge in LLMs.

https://arxiv.org/html/2605.06663v1

It might be possible to train a big generalist that is a composition of modules, some of which can be dropped dynamically at inference time, depending on the prompt.

Re: Local AI needs to be the norm

#605

Post by sinansaka »

I'm betting my startup on it. The subsidised model subscription will start to dry out and providers will lean heavier into locking down how they want their models to be used (Anhropic has been paving the way already). The only way forward is open weight models. If you are working on any LLM powered product be careful betting on utilising user subscriptions.

Re: Local AI needs to be the norm

#606

Post by daveguy »

andybak wrote: post #591
Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

> draw a pretty good pelican on a bike. You mean the famously hard task? The one picked because it stretches frontier models to their limits?

https://simonwillison.net/2026/Apr/22/qwen36-27b/

Maybe this is an example of training overfit. But it won't be too long before local models chew through the "famously hard tasks". Except possibly ARC-AGI. That's one benchmark that is still developing with capabilities. And every time a new ARC-AGI benchmark is released it make the SOTA LLMs look pathetic. Because there is very little understanding or transferability with LLMs. But in terms of benchmark-able micro tasks, the local LLMs are improving.

Re: Local AI needs to be the norm

#607

Post by JamesSwift »

I think moving straight to local models is missing the required next step of open/self-hostable models which is certain to be the "AI future" end-state. Then local models become an optimization on top of that.

I just dont want us to put all this effort in to on-device computation when we need to get to "SOTA-equivalent" self-hosted computation faster.

Re: Local AI needs to be the norm

#608

Post by sanderjd »

henry_kang wrote: post #600
Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

Actually you can do way more things than that. We have optimized it to process 2TB of high def videos on a M5 MBP in under 24 hours, including everything such as speech understanding, face recog, LLM and VLM. Super fun.

Is this project public or have you written about it anywhere?

Re: Local AI needs to be the norm

#609

Post by ge96 »

I'm looking into it since it I'm going to be sending personal info/thoughts would like to keep it local. I have a 4070 running the TheBloke 7B mistral via llama cpp. I still am not using llms daily though other than Google searches.

Re: Local AI needs to be the norm

#610

Post by quantummagic »

andybak wrote: post #591
Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

> draw a pretty good pelican on a bike. You mean the famously hard task? The one picked because it stretches frontier models to their limits?

In fairness, that isn't due to a lack of compute.
Post reply on HN