Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

41–50 of 804 posts

Re: Local AI needs to be the norm

#41

Post by _heimdall »

alex7o wrote: post #37
_heimdall wrote: post #32

Earlier quoted context omitted.

Personally I wouldn't want a couple dozen apps installed all with their own model. It seems easier to have industry specs that define a common interface for local models. I also assume the OS can, or would need to, be involved in proving the models. That may not be a good thing depending on your views of OS vendors, but sharing a single local model does seem more like an OS concern.

I mean the openai API is the industry standard for allowing apps to communicate with models, llama-server has it, oMLX has it, ollama has it, vLLM has it, lmstudio as well. I don't think this is such a hard thing to do, but it requires people to set it up.

I don't know enough about that API surface to know if its a particularly good one for the use cases we'd have, but yes defining a universal spec for all implementors to support wouldn't be a big lift and is done in plenty of other areas already.

Re: Local AI needs to be the norm

#43

Post by karussell »

oytis wrote: post #27
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

> What is the business model of open weight AI?

This is what I do not understand as well and advertising the knowledge and more advanced model is also the only thing that comes to my mind.

Since a month I am using gemma4 locally successfully on a MBP M2 for many search queries (wikipedia style questions) and it is really good, fast enough (30-40t/s) and feels nice as it keeps these queries private. But I don't understand why Google does this and so I think "we" need to find a better solution where the entire pipeline is open and the compute somehow crowdfunded. Because there will be a time when these local models will get more closed like Android is closing down. One restriction they might enforce in the future could be that they cripple the models down for "sensitive" topics like cybersecurity or health topics. Or the government could even feel the need to force them to do so.

Re: Local AI needs to be the norm

#45

Post by jjordan »

Lalabadie wrote: post #21
jjordan wrote: post #6

It feels like we're one technological breakthrough away from all of these data centers going up to be deemed irrelevant.

The cynical take is getting more and more to be the only rational one: The promised mega-data center deals are meant to boost valuations today, not serve tons of customers three years from now.

oof, this bubble popping is gonna be brutal.

Re: Local AI needs to be the norm

#46

Post by Analemma_ »

vb-8448 wrote: post #33

> Use cloud models only when they’re genuinely necessary. The problem is that it's much easier to use the SOTA models (especially if they are subsidized) instead of spending time fixing the knobs with the local one. I just realized this with coding agents, yeah, you probably shouldn't always use latest version at xhigh, but you will end doing it because you do the job in less time, with less "effort" and basically at…

I'm also just not seeing good performance from local models. Every time a thread about LLMs comes up, there are tons of people in the comments insisting that they're getting just as good results from the latest DeepSeek/qwen/whatever as with Opus, and that just hasn't been my experience at all: open-source models just fall over completely compared to Claude when asked to do anything remotely complicated.

I have a sneaking suspicion this is kinda like the situation with Linux in the 90s, where it kinda worked but it reeeeeally wasn't ready for the home user, but you had a lot of people who would insist to your face everything was fine, mostly for ideological reasons.

Re: Local AI needs to be the norm

#48

Post by gdulli »

williamtrask wrote: post #3

I wonder if a popularization moment for local AI will ultimately be the pin-prick that pops the AI bubble. Like the deepseek or openclaw moments but bigger/next.

That's like wondering if enough people discovering local media streaming will disrupt commercial streaming services. It's not going to happen. Most people are not ambitious and will let themselves be controlled by the services of least resistance.

And you can't take comfort in knowing that you, personally, will remain in control of your own computing. The majority will let the range and direction of their thoughts and output be determined by the will of the tech giant whose AI they adopt. And that will shape society.

Re: Local AI needs to be the norm

#49

Post by _heimdall »

alex7o wrote: post #34
bheadmaster wrote: post #28

Earlier quoted context omitted.

> And if you have a local app, how do you take a dependency on whatever random model is installed? Why not ship your own model? In the age of Electron apps, 10GB+ apps are not unheard of.

There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed. Also you can ship 9b fp4 model but it all just depends

Knowing what's installed would have to be an OS API. If LLMs provide a standard API surface to the OS, likely including metadata related to feature support.

Re: Local AI needs to be the norm

#50

Post by aabhay »

TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

Disagree with this. When cost becomes an important factor or the free but worse option becomes compelling and accessible (i.e. on device agent via apple style UX), there has been significant user behavior towards local. Think about stuff like removing backgrounds from photos, OCR on PDFs, who uses paid services for casual usage of these things?
Post reply on HN