Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

251–260 of 804 posts

Re: Local AI needs to be the norm

#251

Post by apsurd »

throwyawayyyy wrote: post #191
giobox wrote: post #155

Earlier quoted context omitted.

> they don't have a used by date For quite a lot of use cases, the current systems arguably do get worse over time if not continually updated. The knowledge cutoff date will start to hurt more and more as the weights age in a hypothetical scenario where you are stuck with them forever. Coding, one of the most popular usescases today, would not be great if it say only understood java to a version from years ago etc. h…

One solution is not to advance anything of course. I'm not even joking, is there going to be a successor to React? I suspect not, with the vast amount of training data for React now, it's going to look silly to move to something else with less support. What is the last new popular programming language, rust? Will there be another one? I suspect not. Same reasoning. The irony of all this AI acceleration talk is it'll…

Humans are notoriously bad at predicting the future. Toward that end, your prediction is laughable. React is the end all be all of UI… lol

Re: Local AI needs to be the norm

#252

Post by codebje »

TheJCDenton wrote: post #140
2ndorderthought wrote: post #54

Earlier quoted context omitted.

Why would you want to try to support all users simple queries on your ai data center if they could run it on their own computer? It builds good will also. it also shows research prowess. For China it's different. They need to show Americans who don't trust them at all because of propaganda that they have no tricks up their sleeve. It also doesn't hurt when Chinese companies drop models for free people can run at home…

Very good point on using local ai to avoid data centers costs. Running AI models on local hardware was exploratory at first, and if it's so easy today it's thanks to open source. It's a little bit coincidental that we have this today, and that mainstream hardware have this capability. The fact that a phone can run very small models is exploratory or some kind of marketing opportunity at best. Why would hardware compa…

I'd expect unified memory architectures (Apple M-series, AMD Ryzen AI series, etc) to be the future of local inference, not GPU cards.

Re: Local AI needs to be the norm

#253

Post by avazhi »

apublicfrog wrote: post #147
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…

> What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time?

Uh… the hardware requirements? And stop acting like some dog shit 8B model the average Joe can run on a laptop is even close to being comparable to what Claude or even Codex can currently do.

I have pretty good hardware and I’ve tinkered with the best sub-150B models you can use and they are awful compared to Anthropic/OAI/Grok.

Re: Local AI needs to be the norm

#254

Post by treis »

Spooky23 wrote: post #245
WarmWash wrote: post #242

Earlier quoted context omitted.

The Chinese labs don't have to make money or be profitable. They are funded by the state to achieve the state's goals, and the global praise of their open models just serves as Chinese soft power. They're state companies, not some kind of ethical VC charity fund project.

Is it so different? If the US’s fascist experiment continues past the current president, we’ll absolutely be nationalizing frontier companies or exerting equivalent control.

Yes, China is very different from the US.

Re: Local AI needs to be the norm

#255

Post by CamperBob2 »

dleslie wrote: post #186
CamperBob2 wrote: post #179

Earlier quoted context omitted.

"Government funding" these days would mean that Trump pays Elon Musk (or more likely vice versa) to make Grok 4.20 the only legal LLM for use by Americans.

Outside of the USA it would not look like a wealth transfer to an oligarch. Not every country is in a crypto-libertarian race to hoard power and wealth.

Not every country is in a crypto-libertarian race to hoard power and wealth.

Meanwhile, in the EU, the model would be collectively financed, trained by a competent, neutral agency... and then completely lobotomized in the name of "the children," "safety," "IP rights," "correct speech," dozens of individual countries' legal and regulatory requirements, and any number of additional vocal, noncontributing NGOs.

So no one would get rich off of the public model, but no one would get much of anything else out of it, either.

As another reply suggests, there's a reason why things happen in the USA first. Even when they don't, the prime movers move here as soon as they can. Or at least they used to.

Re: Local AI needs to be the norm

#256

Post by codebje »

fragmede wrote: post #85
oytis wrote: post #27

Earlier quoted context omitted.

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

The business model is the total lack of attention to Qwen and Kimi that would happen if their models weren't downloadable. Before releasing the weights, there was basically zero attention paid in the western hemisphere to them, for whatever reason. By releasing the weights, they're relevant in the western world. The business model is to get people in the West to pay to use their platform hosting their AI, that otherw…

Baidu have a lot of services I've never heard of, that are highly successful in China. The lack of interest in expanding into Western audiences doesn't seem to matter there - what's different about inference?

Re: Local AI needs to be the norm

#258

Post by pier25 »

pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

How fast do you reckon most people will be able to afford 128-256GB of RAM?

Re: Local AI needs to be the norm

#259

Post by reisse »

winocm wrote: post #247
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

Perhaps I am the odd one out here, but a small part of me wants to see what happens when you run a proprietary SOTA model on a laptop.

Nothing special?

I mean, inference engine might need to get some tweaks, to support whatever compute is available. But then, if you put a few terabytes of disk for swap, and replace RAM to bigger sticks if possible, it should work? Slowly, of course, but there is no reason it should not to.

Re: Local AI needs to be the norm

#260

Post by melagonster »

apsurd wrote: post #251
throwyawayyyy wrote: post #191

Earlier quoted context omitted.

One solution is not to advance anything of course. I'm not even joking, is there going to be a successor to React? I suspect not, with the vast amount of training data for React now, it's going to look silly to move to something else with less support. What is the last new popular programming language, rust? Will there be another one? I suspect not. Same reasoning. The irony of all this AI acceleration talk is it'll…

Humans are notoriously bad at predicting the future. Toward that end, your prediction is laughable. React is the end all be all of UI… lol

Programmers won't be allow to exist in future. Vibe coding is the final resolution people can apply.
Post reply on HN