Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

281–290 of 804 posts

Re: Local AI needs to be the norm

#281

Post by jdub »

manlymuppet wrote: post #215

People are trying to “make the best software”, though. I think the Quixotic accelerationists of AI are more or less a vocal minority of the people who make software, and the choice of online APIs over local systems is largely a choice made for users, rather than developer’s laziness. You can do more and better with private AI today than with local models. There is no getting around that. Even if local AIs get better,…

> Most people won’t settle for a product if it’s not the very best and incredibly convenient.

... uh?

Re: Local AI needs to be the norm

#282

Post by DonsDiscountGas »

antidamage wrote: post #274

The roadblock to this is you seem to have to build it yourself. I've noted that none of the current cloud models are very good at building a replacement for themselves, and there's significant work that needs to be done to make a local LLM reliable in any way. I haven't found a single standalone package that makes setting them up easy. Sure, I can run Hermes Agent and a model, but getting the self-reflection loop in…

Could be an opportunity for a business? Except nobody ever wants to pay for software

Re: Local AI needs to be the norm

#283

Post by ThunderSizzle »

Spooky23 wrote: post #245
WarmWash wrote: post #242

Earlier quoted context omitted.

The Chinese labs don't have to make money or be profitable. They are funded by the state to achieve the state's goals, and the global praise of their open models just serves as Chinese soft power. They're state companies, not some kind of ethical VC charity fund project.

Is it so different? If the US’s fascist experiment continues past the current president, we’ll absolutely be nationalizing frontier companies or exerting equivalent control.

[flagged]

Re: Local AI needs to be the norm

#284

Post by 0xbadcafebee »

Here's some things you can do right now with local models on a consumer device:

- text-to-speech - speech-to-text - dictionary - encyclopedia - help troubleshooting errors - generate common recipes and nutritional facts - proofread emails, blog posts - search a large trove of documents, find information, summarize it (RAG) - manipulate your terminal/browser/etc - analyze a picture or video - generate a picture or video - generate PDFs, documents, etc (code exec) - simple programming - financial analysis/planning - math and science analysis - find simple first aid/medical information - "rubber ducking" but the duck talks back

A quarter of those don't need more than a gig of RAM, the rest benefit from more RAM. Technically you don't even need a GPU, it just makes it faster. I do half that stuff on my laptop with local models every day.

That said, it really doesn't need to be local. I like the idea that I can do all that stuff offline if I'm traveling, but I usually have cell service, and the total tokens is pretty cheap (like $2/month for all my non-coding AI use).

Re: Local AI needs to be the norm

#285

Post by catlifeonmars »

vb-8448 wrote: post #183
lelanthran wrote: post #168

Earlier quoted context omitted.

> My experience is that in medium/big codebases even with single functions going with the xhigh is basically better from a user perspective (faster to get the result, and you can trust it) while going with lower models(e.g. sonnet instead of opus) you have to always carefully review the output because 1 of 10 it will hallucinate, What do you mean "trust it"? It sounds like you want to vibe-code (never look at the out…

With SOTA models I can just set up the instructions (even a little bit fuzzy), go away for 10 or 15 minutes, come back and just check result and adjust when necessary (and most of the time small adjustment are necessary, but the overall work is pretty good). With subpar models I must be more careful on providing instructions and check it step by step because the path it chose is wrong, or I didn't ask for or the agen…

A lot of people aren’t using agents that way. Not saying that it’s not a legitimate use or anything, just that I think the use cases are different. And yeah maybe for your specific use case, sota hosted models are the right choice

Re: Local AI needs to be the norm

#286

Post by selcuka »

ai_fry_ur_brain wrote: post #229
apublicfrog wrote: post #147

Earlier quoted context omitted.

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…

95% of usecases. What are you smoking.

There are very good open weight models (such as DeepSeek v4 Flash) that can run on consumer level hardware.

Note that we are talking about 95% of everyone's use cases, not your specific use cases (which could require better models all the time).

Re: Local AI needs to be the norm

#287

Post by elbasti »

pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

> The question will be: how much of the current compute capacity craze will local hosting give the kiss of death to and what that means for the market.

This will depend on how much inference happens for consumer (desktop, local) vs enterprise ("cloud"), vs consumer mobile (probably also cloud).

I would assume that the proportion of "consumer, local" is small relative to enterprise and mobile.

Re: Local AI needs to be the norm

#288

Post by selcuka »

sroussey wrote: post #142
js8 wrote: post #89

Earlier quoted context omitted.

What is the business model of Wikipedia? I don't think there is any. Not everything good in our society needs to have a "business model". People still work on it. It's FINE.

> What is the business model of Wikipedia? Donations. Have you donated lately? Wikipedia is cheap compared to creating and training models. I don’t think donations will suffice at all. As an example, we had millions of web developers download and install Firebug before browsers shipped their own dev tools. Donations over the course of multiple years would have paid my salary for a month if I were not a volunteer. But…

> Wikipedia is cheap compared to creating and training models.

DeepSeek said it spent $5.6M [1] on training V3, which doesn't sound too much for a near-SOTA model.

An open source entity can come up with a hybrid business model, such as requiring a small fee from those who want to host the model as a business for the first n months following the release of a new model, but making it fully free for individuals.

[1] https://arxiv.org/pdf/2412.19437

Re: Local AI needs to be the norm

#289

Post by CamperBob2 »

ai_fry_ur_brain wrote: post #213
zozbot234 wrote: post #201

Earlier quoted context omitted.

> I don't think cloud models are going away; the hardware for good perf is expensive I think local AI will win in its niche by repurposing users' existing hardware, especially as cloud hardware itself gets increasingly bottlenecked in all sorts of ways and the price of cloud tokens rises. You don't have to care about "bad" performance when you've got dedicated hardware that runs your workloads 24/7. Time-critical wor…

[flagged]

Other sites beckon.

Re: Local AI needs to be the norm

#290

Post by root_axis »

pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model's reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.
Post reply on HN