Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

141–150 of 804 posts

Re: Local AI needs to be the norm

#141

Post by lelanthran »

krupan wrote: post #105
intothemild wrote: post #30

Earlier quoted context omitted.

That's already happening. Qwen3.6 and Gemma4. Basically small and medium models that are crazy well trained for their sizes. Then we have a lot of specular decoding stuff like MTP and others coming to speed up responses, and finally better quantisation to use less memory. Local LLM is the future, and the larger labs know that the open models will eat their lunch once people realise that the gap is only a few months.…

And how were those models developed and trained?

> And how were those models developed and trained?

That's irrelevant to my decision to use local or not.

Re: Local AI needs to be the norm

#142

Post by sroussey »

js8 wrote: post #89
oytis wrote: post #27

Earlier quoted context omitted.

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

What is the business model of Wikipedia? I don't think there is any. Not everything good in our society needs to have a "business model". People still work on it. It's FINE.

> What is the business model of Wikipedia?

Donations. Have you donated lately?

Wikipedia is cheap compared to creating and training models.

I don’t think donations will suffice at all.

As an example, we had millions of web developers download and install Firebug before browsers shipped their own dev tools. Donations over the course of multiple years would have paid my salary for a month if I were not a volunteer.

But from the “it’s fine” point of view, models will be baked into your OS.

Then later models will be embedded into hardware. Likely only OS makers models.

Re: Local AI needs to be the norm

#143

Post by vb-8448 »

lelanthran wrote: post #124
vb-8448 wrote: post #33

> Use cloud models only when they’re genuinely necessary. The problem is that it's much easier to use the SOTA models (especially if they are subsidized) instead of spending time fixing the knobs with the local one. I just realized this with coding agents, yeah, you probably shouldn't always use latest version at xhigh, but you will end doing it because you do the job in less time, with less "effort" and basically at…

> The problem is that it's much easier to use the SOTA models (especially if they are subsidized) instead of spending time fixing the knobs with the local one. That's not a problem, that's a feature ; I have something like 8 tabs open to different free-tier providers. ChatGPT, Claude and Gemini are the SOTA ones. I have no problem maxing one out, then moving to the next. I can do this all day, have them implement spe…

We are speaking about local AI, and having all this SOTA models basically for free is blocking the progress of local or independent third party setups.

Re: Local AI needs to be the norm

#144

Post by [deleted] »

wmf wrote: post #95
bytecauldron wrote: post #71

Earlier quoted context omitted.

This is a bit disingenuous. People aren't losing their shit about a local model being installed. It's the lack of user autonomy. Just give the option to download a model instead of a silent install. It's not that hard. This is how every other local option works.

AFAIK Apple and MS auto-download local models.

[deleted]

Re: Local AI needs to be the norm

#147

Post by apublicfrog »

TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control.

What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next tier that comes out, but the impact of that is very low.

Re: Local AI needs to be the norm

#148

Post by crazygringo »

themafia wrote: post #75
StilesCrisis wrote: post #68

Earlier quoted context omitted.

I'm guessing you immediately close the What's New Chrome tab when you update? https://developer.chrome.com/blog/new-in-chrome-148#prompt-a... https://www.google.com/chrome/ai-innovations/ They have absolutely not been shy about any of this.

I've never had a "What's new" tab ever open because I disable the customized home page where that's displayed. I'm guessing you're not aware that's an option. Please show me where in either of those documents it explains it's going to download a 4GB model.

I use an extension that gives me a customized homepage, but I still always get the "what's new" tab on every major version upgrade.

It's a totally separate tab that opens. It's got nothing to do with what you use as your homepage.

Re: Local AI needs to be the norm

#149

Post by CamperBob2 »

iLoveOncall wrote: post #126
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

The mainstream audience does not have the faintest idea that "local AI" is even a thing.

Just as their counterparts in 1975 had no idea that "personal computers" were even a thing.

Read through a 1970s-era issue of Popular Electronics or Byte, and then spend some time surfing /r/LocalLlama. You'll get a sense of real-time deja vu, like you're watching history unfold again.

Re: Local AI needs to be the norm

#150

Post by vb-8448 »

lelanthran wrote: post #133
Analemma_ wrote: post #46

Earlier quoted context omitted.

I'm also just not seeing good performance from local models. Every time a thread about LLMs comes up, there are tons of people in the comments insisting that they're getting just as good results from the latest DeepSeek/qwen/whatever as with Opus, and that just hasn't been my experience at all: open-source models just fall over completely compared to Claude when asked to do anything remotely complicated. I have a sne…

> Every time a thread about LLMs comes up, there are tons of people in the comments insisting that they're getting just as good results from the latest DeepSeek/qwen/whatever as with Opus, and that just hasn't been my experience at all: open-source models just fall over completely compared to Claude when asked to do anything remotely complicated. Different usage patterns - you want to issue a single spec then walk aw…

My experience is that in medium/big codebases even with single functions going with the xhigh is basically better from a user perspective (faster to get the result, and you can trust it) while going with lower models(e.g. sonnet instead of opus) you have to always carefully review the output because 1 of 10 it will hallucinate, you won't catch it immediately and at some point it will bite you.
Post reply on HN