Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

91–100 of 804 posts

Re: Local AI needs to be the norm

#91

Post by dleslie »

oytis wrote: post #27
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

This is where government funding can play a role.

Sometimes there are things where the public good is best served with public expenditure.

Re: Local AI needs to be the norm

#92

Post by scriptsmith »

I've got some demos of what the new Prompt API in Chrome that uses a local model can do: https://adsm.dev/posts/prompt-api/#what-could-you-build-with...

As OP says, it shines in constrained environments where the model is transforming user-owned data. Definitely less useful for anything more open-ended.

Re: Local AI needs to be the norm

#93

Post by refulgentis »

The shitty thing here is, either everyone's shipping 800 MB at least with their binary, or, you have to rely on the platform vendor anyway. I'm hoping there's enough external pressure that the OS vendors turn it more into a repository than a blessed-model-garden.

Re: Local AI needs to be the norm

#94

Post by za_creature »

StilesCrisis wrote: post #70
ekjhgkejhgk wrote: post #53

Earlier quoted context omitted.

You don't understand the difference between "I run a local LLM because I chose to" vs "The browser chose to run a local LLM and I have no say"? You don't understand? Not to mention that the LLM that I choose to run requires a monster machine and is infinitely more capable than whatever google chose to put on their browser? I mean, none of this affects me because I don't use chrome, obviously, but you don't see the di…

Did you opt into WebGPU? QUIC? Canvas 2D? Brotli? Browsers don't work that way.

The size difference between the local LLM and all of the above is about... the size of the local LLM.

Re: Local AI needs to be the norm

#95

Post by wmf »

bytecauldron wrote: post #71
mattlondon wrote: post #47

Yet there is another post a few rows down where people are losing their shit that Chrome has a local LLM model that uses a couple of GB of space for local-inference. Damned if they do, damned if they don't.

This is a bit disingenuous. People aren't losing their shit about a local model being installed. It's the lack of user autonomy. Just give the option to download a model instead of a silent install. It's not that hard. This is how every other local option works.

AFAIK Apple and MS auto-download local models.

Re: Local AI needs to be the norm

#96

Post by williamtrask »

gdulli wrote: post #48
williamtrask wrote: post #3

I wonder if a popularization moment for local AI will ultimately be the pin-prick that pops the AI bubble. Like the deepseek or openclaw moments but bigger/next.

That's like wondering if enough people discovering local media streaming will disrupt commercial streaming services. It's not going to happen. Most people are not ambitious and will let themselves be controlled by the services of least resistance. And you can't take comfort in knowing that you, personally, will remain in control of your own computing. The majority will let the range and direction of their thoughts an…

Yeah... probably right. I do hold out hope that this is mostly a timeframe thing. Like, the library, printing press, etc. all had their moments of centralization. But eventually they federated.

Re: Local AI needs to be the norm

#97

Post by 2ndorderthought »

scriptsmith wrote: post #92

I've got some demos of what the new Prompt API in Chrome that uses a local model can do: https://adsm.dev/posts/prompt-api/#what-could-you-build-with... As OP says, it shines in constrained environments where the model is transforming user-owned data. Definitely less useful for anything more open-ended.

Yea I do not recommend treating chromes prompt API as a good example of local LLMs. It's fine and stuff but it's really weak. 8b models from a year ago are better in some ways. And a lot of the recent model drops are meaningfully better.

Re: Local AI needs to be the norm

#98

Post by daishi55 »

> We are building applications that stop working the moment the server crashes or a credit card expires

Isn’t this true of any application that accesses anything not running on your computer? This is just describing what it means to add an API call to your app. Nothing to do with AI (?)

Re: Local AI needs to be the norm

#99

Post by krupan »

kibwen wrote: post #26
timeattack wrote: post #9

My problem with LLMs (apart from philosophical aspects and economical impact) is that it would be unlikely for any of us to be able to train something functional locally (toy-like LLMs -- sure, but something really useful -- no). Apart from that it requires immense computing power, it also requires a dataset which is for the most part is obtained illegally.

This seems overly pessimistic. I may personally be of modest intelligence, but to acquire the intelligence that I do have, I did not need to train on every book ever written, every Wikipedia article ever written, every blog post ever written, every reference manual ever written, every line of code ever written, and so on. In fact, I didn't train on even 1% of those materials, or even 0.00000000001% of those. The text…

What does this even have to do with the parent? Your capabilities have nothing to do with LLM capabilities. The two work in completely different ways. The reason LLMs work is because they are huge and have been trained on vast amounts of data, full stop. Sure, there's potential someday to get something useful using less data, but we aren't there.

Re: Local AI needs to be the norm

#100

Post by hparadiz »

dakolli wrote: post #90
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. To sell tokens profitably you'd need to be able to run inference at 150 tokens per second for less than $1,000 USD a month. I don't think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized. You can only squeeze so many parameters on consumer grade…

Posts like this are so funny to me. I'm staring at a mountain of old hardware right now that cost about $20k ten years ago. I have to pay someone now to come haul it away. What makes you think the current new hardware won't end up with the same fate.

> Just write your own fkin code people

Bro is nostalgic for googling random stack overflow threads for 10 days to figure out a bug the agent fixes in an hour.

Post reply on HN