Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

791–800 of 804 posts

Re: Local AI needs to be the norm

#791

Post by henry_kang »

nullc wrote: post #232
giobox wrote: post #155

Earlier quoted context omitted.

> they don't have a used by date For quite a lot of use cases, the current systems arguably do get worse over time if not continually updated. The knowledge cutoff date will start to hurt more and more as the weights age in a hypothetical scenario where you are stuck with them forever. Coding, one of the most popular usescases today, would not be great if it say only understood java to a version from years ago etc. h…

Small models are more useful for "doing stuff" than "knowing stuff" to begin with. Add in an agentic harness and a small model can happily read more current information on demand (including from e.g. a local wikipedia snapshot).

This feels increasingly true.

A lot of useful AI work is shifting from “knowing more” to “working with more context”, files, recordings, repos, screenshots, browsing history, etc.

Once that happens, memory and orchestration start mattering much more than raw model size.

Re: Local AI needs to be the norm

#792

Post by a96 »

krupan wrote: post #112

If you don't need a lot of smarts, do you even need an LLM? Aren't older machine learning techniques just as good, or like, you know, old-school algorithms?

Yes. For essentially any problem where a complete solution exists that doesn't use an LLM, it will beat any solution that does in size, speed, energy use, reliability and everything else.

Naturally, it's actually complicated. But LLM is a considerable weight and risk. Maybe it's worth involving and maybe not.

Re: Local AI needs to be the norm

#793

Post by DrewADesign »

zozbot234 wrote: post #788
DrewADesign wrote: post #784

Earlier quoted context omitted.

If those data centers were good enough, they’d save themselves a few billion dollars and just do more of the same, wouldn’t they? Many current video games struggle on the 3060— it’s like 10 times slower for interference than a 4090 even. They’re reintroducing it because their upstream business of selling brand new insanely expensive GPUs required for every new data center is making it impossible for people to buy GPU…

The new GPUs are a lot better than the old ones, to be sure. They're also a whole lot harder to get ahold of in quantity. That's no different than the reason for officially reintroducing the 3060.

That doesn’t make this more sustainable or viable in the long term, which is the entire point

Re: Local AI needs to be the norm

#794

Post by reissbaker »

zozbot234 wrote: post #738
reissbaker wrote: post #721

Earlier quoted context omitted.

The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models. Within any time budget, you can get many orders of magnitude more large-model tokens off an 8xB200 than off a local rig. Therefore cloud tokens have a huge capital efficiency advantage over local rigs. That will continue basically forever, since there will always be large cloud companies…

> The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models. Sometimes it really is free though, because the hardware was bought to serve some other existing needs and that capital expense was fully depreciated quite some time ago. Underutilised hardware is essentially ubiquitous. > Within any time budget, you can get many orders of magnitude mor…

> Sometimes it really is free though, because the hardware was bought to serve some other existing needs and that capital expense was fully depreciated quite some time ago.

No one has 8xRTX Pro 6000s that have depreciated to zero "quite some time ago."

> But using that 8xB200 setup to run inference in cheap, non-frontier models is plain waste

From whose perspective? If someone wants to run an open-source model — and plenty do — someone buying or renting an 8xB200 to serve it cheaply at scale is much better than everyone buying huge amounts of pointless, wasted hardware such as 8xRTX Pro 6000s for $80,000 per person.

Re: Local AI needs to be the norm

#795

Post by reissbaker »

otabdeveloper4 wrote: post #762
reissbaker wrote: post #718

Earlier quoted context omitted.

There already are many subscriptions for LLM tokens: OpenAI, Claude, Synthetic (shameless plug), Zai... I'm not sure what you mean by "There will not ever be a monthly subscription for LLM tokens." That already exists!

Monthly subscriptions is a "first hit is free" promo. In the future LLMs will be priced per token, not all-you-can-eat.

LLM subscriptions are not "all you can eat," they have rate limits — and fundamentally there is no difference between subscription-with-rate-limit and typical usage-based business practices. Subscriptions are simply usage-based pricing with volume discounts in exchange for upfront payment; every single usage-based provider of pretty much anything offers the same kind of discounting for buying volume commitments upfront. Although from a business sense, subscriptions are even better than volume discounts... Because they're recurring, whereas reserved volume might not recur.

Subscriptions aren't gonna go away. They're great for businesses. Rate limits or pricing might change but the underlying business model is very good.

The reason usage-based is so much more expensive than subscription isn't that usage-based is the "true" cost and subscription is a loss leader — just like a buying 30 consecutive day passes to a gym being more expensive than a monthly membership isn't a result of memberships being a loss leader. Memberships are the business model! The day passes are overpriced to steer you into buying the membership.

Re: Local AI needs to be the norm

#796

Post by simooooo »

pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

Even on a 5090 qwen is really impressive. Felt as good as Claude for little projects.

Re: Local AI needs to be the norm

#799

Post by ThunderSizzle »

Aurornis wrote: post #321
adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

I experiment a lot with local models, and I agree. I have a lot of fun with the local models and seeing what they can do. I appreciate the SOTA models even more after my local experiments. The local models are really impressive these days, but the gap to SOTA is huge for complex tasks.

What if you split it into less complex tasks? E.g. use the model to help decompose the task into parts, then help it iterate through it.

Gives you more control over the outcome and more steering anyway.

Re: Local AI needs to be the norm

#800

Post by digitaltrees »

robrenaud wrote: post #604
epicureanideal wrote: post #369

Earlier quoted context omitted.

I also wonder about JS only, Python only, etc models. Maybe the future is a selection of local, specific stack trained models?

There is some recent work on modularizing knowledge in LLMs. https://arxiv.org/html/2605.06663v1 It might be possible to train a big generalist that is a composition of modules, some of which can be dropped dynamically at inference time, depending on the prompt.

Cool. Thanks for sharing. I am thinking about creating a series of smaller models for specific purposes and then orchestrating them so they mirror the human brain which is a bunch of subsystems that give multiple opinions about the same stimulus
Post reply on HN