Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

391–400 of 804 posts

Re: Local AI needs to be the norm

#391

Post by janalsncm »

rmunn wrote: post #366

For image generation, this has already happened. To what degree, I can't tell, as I don't do image generation much so I don't have numbers on Midjourney subscriptions or any other image-AI-as-a-service sites. But civitai.com has become a place where people share their models, based off of Stable Diffusion or other similar bases, with various fine-tunings to achieve desired results. You name it, you can find a model f…

> I haven't seen a text-based model sharing site spring up yet (perhaps they already have and I don't know about it yet)

Huggingface.

The reason HF doesn’t also compete for image gen is probably some combination of momentum from Civit AI and HF not wanting to deal with the moderation headache.

Re: Local AI needs to be the norm

#392

Post by stubish »

root_axis wrote: post #290
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model's reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.

It depends on what you mean for 'productive'. Article mainly seems to be about targeting consumer level hardware, such as the Neural Processing Unit you need for a 'Copilot PC'. Windows Recall is (was?) one such local AI application. If Microsoft get their way and my next PC has one, I look forward to using it for 'productive' purposes such as playing games, handling natural language stuff and leaving my GPU free for GPUing.

Re: Local AI needs to be the norm

#393

Post by emadb »

pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

Do you think small models will arrive? I mean if I need to write a web application in typescript why should I use a model that knows all the programming languages and it is able to reply to any questions about almost everything? I just a need a small performant model that knows how to write web applications in typescript. That could be very helpful and easy to run on my laptop.

Re: Local AI needs to be the norm

#394

Post by otabdeveloper4 »

reissbaker wrote: post #336
zozbot234 wrote: post #201

Earlier quoted context omitted.

> I don't think cloud models are going away; the hardware for good perf is expensive I think local AI will win in its niche by repurposing users' existing hardware, especially as cloud hardware itself gets increasingly bottlenecked in all sorts of ways and the price of cloud tokens rises. You don't have to care about "bad" performance when you've got dedicated hardware that runs your workloads 24/7. Time-critical wor…

Users do not have an existing $80k of hardware, are not going to buy $80k of hardware for worse performance than paying $100/month, and models are continuing to grow in size while memory grows in price.

> paying $100/month

There will not ever be a monthly subscription for LLM tokens. The economics isn't there.

Local tokens will always be cheaper.

Re: Local AI needs to be the norm

#395

Post by DeathArrow »

tuananh wrote: post #319
binyu wrote: post #309

Earlier quoted context omitted.

DeepSeek V4 with 1 million token context window is pretty powerful, although still not there. There's hope that Opus 4.5 level performance locally is not that far away.

From what I read, ds v4 is very close with opus 4.6 performance.

The full model is, not the quantized versions.

Re: Local AI needs to be the norm

#396

Post by DeathArrow »

bugglebeetle wrote: post #316
adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

The frontier Chinese open source models are already at this level, GLM-5.1 and Kimi K2.6 specifically.

But you can't run the locally at full quality. And quantized versions you can run locally are a far cry from Opus 4.6.

Re: Local AI needs to be the norm

#397

Post by thot_experiment »

emadb wrote: post #393
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

Do you think small models will arrive? I mean if I need to write a web application in typescript why should I use a model that knows all the programming languages and it is able to reply to any questions about almost everything? I just a need a small performant model that knows how to write web applications in typescript. That could be very helpful and easy to run on my laptop.

Depending on your laptop, if your laptop is a Strix Halo or a Macbook with a decent amount of ram, that day they arrived is about 6 months ago, and today if you can run Gemma 31b, you're golden for your basic workslop code. You can do most of it with local models. Heck, for a lot of the tier of programming you might encounter in the average job Qwen 35b MoE is good enough and it can hit 100tok/s on decent hardware.

Re: Local AI needs to be the norm

#398

Post by [deleted] »

adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

[deleted]

Re: Local AI needs to be the norm

#399

Post by otabdeveloper4 »

ai_fry_ur_brain wrote: post #216
nullc wrote: post #175

Earlier quoted context omitted.

Here is an example-- I'm running hermes + qwen3.6-27b on a workstation GPU (an older RTX A6000 which gets 55tok/s, though people run this model on more limited hardware). A friend an I had previously worked on an entropy extraction scheme and he recently got around to making a writeup about our work: https://wuille.net/posts/binomial-randomness-extractors/ I instructed the agent to read the URL, implement the techniq…

This is maybe the first time Ive seen someone claim to do something useful with such a small model. Congrats, but you're in the 0.0001% thats not just frying their brains, fapping to their local models or doing various magic tricks like a toddler entertained by playing with velcro. At the end of the day you lost an opportunity to improve yourself and excercise your brain, maybe the opportunity cost is worth it idk, b…

Sounds like you're coping for the vendor lock-in you cornered yourself into.

Re: Local AI needs to be the norm

#400

Post by bugglebeetle »

DeathArrow wrote: post #396
bugglebeetle wrote: post #316

Earlier quoted context omitted.

The frontier Chinese open source models are already at this level, GLM-5.1 and Kimi K2.6 specifically.

But you can't run the locally at full quality. And quantized versions you can run locally are a far cry from Opus 4.6.

Anthropic serves quantized versions of their models and you can run q8 locally.
Post reply on HN