Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

501–510 of 804 posts

Re: Local AI needs to be the norm

#501

Post by fennecfoxy »

reisse wrote: post #212
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

>It's here, right now.

I mean I've been forcing my good old 1080ti to run local models since a short while after llama was first leaked.

But I wouldn't say "local models are here" in the same way as "year of the Linux desktop!111"

Until someone can just go out and buy some sort of "AI pod" that they can take home, plug in and hit one button on a mobile app to select a model (or even just hide models behind various personas) then I wouldn't say it's quite there yet.

It's important that the average consumer can do it, I think the limitations for that are: things are changing too quickly, ram+compute components are exceedingly expensive now, we're still waiting on better controls/harnesses for this stuff to stop consumers not just from shooting themselves in the foot, but blowing their foot clean off.

Would be interesting to see a Taalas-like chip in a product, albeit there's so many changes going on atm with diffusion based models, Google's Turboquant (which as someone who has had to almost always run quantized models, makes a lot of sense to me).

Re: Local AI needs to be the norm

#502

Post by DrScientist »

pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

I think it's inevitable that access to good enough LLM models will be democratised.

However that's not the real battle here. The real battle is control of information to operate over.

While I might have access to a decent model - I don't have the huge integrated databases of everything that companies like Google have, and increasingly governments will accumulate.

As a citizen AI operating of these large datasets is where the concern should be.

Re: Local AI needs to be the norm

#503

Post by fennecfoxy »

nullsanity wrote: post #337
yieldcrv wrote: post #312

Earlier quoted context omitted.

I need to see these proper harnesses I tried oMLX and OpenCode a few weeks ago and the 65k context window was useless, it tried to analyze a very small codebase before going full on agentic and ran out of context window immediately I don't have time to tweak 1,000 permutations of settings just re-prove that its not as smart as Opus 4.6 I need out the box multimodal behavior as similar as typing claude in the command…

Hey man, you can just say "I'm lazy, so I'm staying with the cloud. if I wanted to use my brain, I wouldn't be using AI, gosh" - it's much shorter.

Personal attacks are against the rules, by the way.

Re: Local AI needs to be the norm

#504

Post by gpugreg »

almogodel wrote: post #481

Remember nodes and graphs? A comfy user interface allows pretty incredible wiring among models local ai is like eurorack. The current graph skews heavily towards a a pair of small dense models collaborating with the large heavyweights selectively. It’s Qwen 3.6 27B with Gemma 4 31B, both unquantized, bf16/fp16, with phi 14b, nemotron cascade 2, and then those large heavyweights, r1 and subsequent deepseek models incl…

> Slow, resource intense, better than non local ai

Why should connecting small models to big models result in higher output quality than just running the big models without the small models?

Re: Local AI needs to be the norm

#505

Post by fennecfoxy »

Schiendelman wrote: post #267
pier25 wrote: post #258

Earlier quoted context omitted.

How fast do you reckon most people will be able to afford 128-256GB of RAM?

Other than this recent spike, it's been trending cheaper continuously for decades. In a few years 128GB will be as affordable as 12GB (what flagship phones have now) is today.

Nope.

Because late stage capitalism demands endless growth in order to pay executives and shareholders (especially those late to the train) more and more YoY.

And those requirements for growth mean that cost cutting is needed. Over the past few decades cost _have_ been cut, building things more efficiently, components becoming cheaper, larger volumes in mass manufacturing.

But we have already reached a point where there are no other places to cut than the quality of the product itself. Look to shrinkflation in food and other places - look at how "live action" versions are being made of previously animated movies, how game franchises from 2 decades ago are being brought back from the dead, the huge influx of remasters etc.

Why? Because it's cheaper to revive/reuse an existing IP than it is to create a new one + it guarantees success with the drooling consumer masses. And cheaper = more Ferraris for the multi millionaire/billionaire execs.

See how much Mario movie made? Just wait...bet you there'll be a live action version. ;)

Re: Local AI needs to be the norm

#506

Post by thot_experiment »

cbg0 wrote: post #486
thot_experiment wrote: post #466

Earlier quoted context omitted.

Benchmarks only give you the roughest idea of how models compare in real world use. They're essentially useless beyond maybe classifying models into a few buckets. The only way you gain an understanding of something as complex as how an LLM integrates with your workflow is by doing it and measuring across many trials. I've been running Opus 4.7 in Claude Code and Gemma 4 31b in parallel on projects for hours a day th…

"essentially useless" is a gross overstatement. Your personal benchmarks will always provide you with the most value, but disregarding standardized benchmarks because you care more about vibes is not exactly scientific.

Sorry, "essentially useless in the context of local model availability". It's a fine model but it's tier of inference is fully fungible.

Re: Local AI needs to be the norm

#507

Post by Hamuko »

pbgcp2026 wrote: post #492
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

I'm sorry to spoil it for you, but Perl script was able to do all of that like ... 10 years ago? The out-of-the-box Shotwell manages photos quite well without any intelligence. The problem, as people mentioned above, is SOTA models cognitive and tooling abilities. Also, have you noticed as top-end Mac Studios got downgraded recently? They don't want you to have access to frontier models. And you will not have it. See…

>Also, have you noticed as top-end Mac Studios got downgraded recently? They don't want you to have access to frontier models. And you will not have it.

Isn't that a function of RAM supply not being available now?

Re: Local AI needs to be the norm

#508

Post by ubercore »

pbgcp2026 wrote: post #492
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

I'm sorry to spoil it for you, but Perl script was able to do all of that like ... 10 years ago? The out-of-the-box Shotwell manages photos quite well without any intelligence. The problem, as people mentioned above, is SOTA models cognitive and tooling abilities. Also, have you noticed as top-end Mac Studios got downgraded recently? They don't want you to have access to frontier models. And you will not have it. See…

The conspiracy angle here is not really relevant. Ram is expensive and they're gearing up for M5 studios. Not the illuminati keeping better LLM models out of your hands.

Re: Local AI needs to be the norm

#509

Post by Galanwe »

root_axis wrote: post #351
wincy wrote: post #317

Earlier quoted context omitted.

Cool, thanks for the information. I guess they drive prices down by massively parallelizing requests on say an H100 X8 array? So this is spread across. So if I say, wanted to use it for 8 hours a day in my theoretical world it’d be too expensive. My work definitely wouldn’t pay $100,000 for a server farm even if it’d give an AI to all our employees, you’d have to have engineers, a colocation space, basically all the…

Well $100k was a generous guesstimate for some time in the future where something like an Opus 4.7 is old news. If we think about the near future, something like Kimi2.6 is within the realm of Opus 4.6 today, but requires closer to $700k in hardware to run.

Kimi 2.6 is very close to the Opus family from my experience. Also it does absolutely not require $700k to be able to run locally in an interactive fashion. We are talking more in the range of $10k for a slow Q2 with degraded perplexity, to ~$35k for an acceptably fast 200k context Q4 (quasi lossless perplexity).

Re: Local AI needs to be the norm

#510

Post by Fokamul »

manmal wrote: post #475
digitaltrees wrote: post #269

Earlier quoted context omitted.

I built my own IDE and run my own model specifically to have private agentic coding. I can still access model APIs but I can be purely local if I want too. It’s amazing.

Curious, why did Zed with ACP not work for you?

I'm just guessing, but IDE which is using 3D acceleration just for stupid UI to run "smoothly", that is ridiculous.

Who runs IDE with LLM agents accessing your local filesystem, on bare metal?

Or am I alone to run everything LLM related on my VM just for development work. Then because of ZED genius decision, you need to share your GPU to VM, then some important features will not work, like snapshots. So you also need workaround for this, etc.

Too much hassle, Zed is not for me.

But I'm anti-Apple, so maybe that's the reason :)

Btw, even "ImHex" devs realized this and they're providing version without acceleration for VM use. They're using ImGui. Using it for local desktop app UI is also ridiculous, imho. Whatever.

Post reply on HN