Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

31–40 of 804 posts

Re: Local AI needs to be the norm

#31

Post by worldsayshi »

oytis wrote: post #27
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

It should be feasible to crowd fund training runs right?

Re: Local AI needs to be the norm

#32

Post by _heimdall »

bheadmaster wrote: post #28
barrkel wrote: post #25

Local models are extraordinarily expensive if you're not maximizing throughput, and you're not going to be maximizing it. Local models need to be resident in expensive RAM, the kind that has fat pipes to compute. And if you have a local app, how do you take a dependency on whatever random model is installed? Does it support your tool calling complexity? Does it have multimodal input? Does it support system messages i…

> And if you have a local app, how do you take a dependency on whatever random model is installed? Why not ship your own model? In the age of Electron apps, 10GB+ apps are not unheard of.

Personally I wouldn't want a couple dozen apps installed all with their own model.

It seems easier to have industry specs that define a common interface for local models.

I also assume the OS can, or would need to, be involved in proving the models. That may not be a good thing depending on your views of OS vendors, but sharing a single local model does seem more like an OS concern.

Re: Local AI needs to be the norm

#33

Post by vb-8448 »

> Use cloud models only when they’re genuinely necessary.

The problem is that it's much easier to use the SOTA models (especially if they are subsidized) instead of spending time fixing the knobs with the local one.

I just realized this with coding agents, yeah, you probably shouldn't always use latest version at xhigh, but you will end doing it because you do the job in less time, with less "effort" and basically at the same price.

I guess we'll see a real effort for local AI only when major vendors will start billing based on actual token usage.

Re: Local AI needs to be the norm

#34

Post by alex7o »

bheadmaster wrote: post #28
barrkel wrote: post #25

Local models are extraordinarily expensive if you're not maximizing throughput, and you're not going to be maximizing it. Local models need to be resident in expensive RAM, the kind that has fat pipes to compute. And if you have a local app, how do you take a dependency on whatever random model is installed? Does it support your tool calling complexity? Does it have multimodal input? Does it support system messages i…

> And if you have a local app, how do you take a dependency on whatever random model is installed? Why not ship your own model? In the age of Electron apps, 10GB+ apps are not unheard of.

There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed. Also you can ship 9b fp4 model but it all just depends

Re: Local AI needs to be the norm

#35

Post by LPisGood »

barrkel wrote: post #25

Local models are extraordinarily expensive if you're not maximizing throughput, and you're not going to be maximizing it. Local models need to be resident in expensive RAM, the kind that has fat pipes to compute. And if you have a local app, how do you take a dependency on whatever random model is installed? Does it support your tool calling complexity? Does it have multimodal input? Does it support system messages i…

> And if you have a local app, how do you take a dependency on whatever random model is installed?

Reading the tea leaves here, it will probably be common for OS’s to have built in models that can be accessed via API. Apple already does this.

Re: Local AI needs to be the norm

#36

Post by LPisGood »

alex7o wrote: post #34
bheadmaster wrote: post #28

Earlier quoted context omitted.

> And if you have a local app, how do you take a dependency on whatever random model is installed? Why not ship your own model? In the age of Electron apps, 10GB+ apps are not unheard of.

There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed. Also you can ship 9b fp4 model but it all just depends

You can know what the user has installed if the OS developer offers something.

Re: Local AI needs to be the norm

#37

Post by alex7o »

_heimdall wrote: post #32
bheadmaster wrote: post #28

Earlier quoted context omitted.

> And if you have a local app, how do you take a dependency on whatever random model is installed? Why not ship your own model? In the age of Electron apps, 10GB+ apps are not unheard of.

Personally I wouldn't want a couple dozen apps installed all with their own model. It seems easier to have industry specs that define a common interface for local models. I also assume the OS can, or would need to, be involved in proving the models. That may not be a good thing depending on your views of OS vendors, but sharing a single local model does seem more like an OS concern.

I mean the openai API is the industry standard for allowing apps to communicate with models, llama-server has it, oMLX has it, ollama has it, vLLM has it, lmstudio as well. I don't think this is such a hard thing to do, but it requires people to set it up.

Re: Local AI needs to be the norm

#38

Post by _heimdall »

kibwen wrote: post #26
timeattack wrote: post #9

My problem with LLMs (apart from philosophical aspects and economical impact) is that it would be unlikely for any of us to be able to train something functional locally (toy-like LLMs -- sure, but something really useful -- no). Apart from that it requires immense computing power, it also requires a dataset which is for the most part is obtained illegally.

This seems overly pessimistic. I may personally be of modest intelligence, but to acquire the intelligence that I do have, I did not need to train on every book ever written, every Wikipedia article ever written, every blog post ever written, every reference manual ever written, every line of code ever written, and so on. In fact, I didn't train on even 1% of those materials, or even 0.00000000001% of those. The text…

You're also embodied and experiencing the world around you with more senses than only the ability to read text.

Re: Local AI needs to be the norm

#39

Post by PAndreew »

oytis wrote: post #27
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

Perhaps you can create a compelling UX around it and sell it as a subscription. "Normies" will not be able/willing to build it. You can then patch the model/ship new features around it as it evolves. For example I have built an ambient todo list / health data extractor using Gemma 4 2EB and Whisper. Nothing to brag about but it does fairly decent job even in foreign languages.

Re: Local AI needs to be the norm

#40

Post by YZF »

Waterluvian wrote: post #8
i_love_retros wrote: post #7

Earlier quoted context omitted.

What would that breakthrough be?

Magic math and computer science that allows us to get the same quality response for a fraction of the GPU.

The current LLMs are also "magic" so anything is possible. AFAIK there is no proof that the current architecture is optimal. And we have our brains as a pretty powerful local thinking machine as a counter-example to the idea that thinking has to happen in data centers.
Post reply on HN