Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

651–660 of 804 posts

Re: Local AI needs to be the norm

#651

Post by zozbot234 »

fleventynine wrote: post #647
r0b05 wrote: post #404

Earlier quoted context omitted.

It's an interesting point but local gpu efficiency is not something I think about when I'm being rate limited or when my subscription costs keep rising.

I think folks in this thread are underestimating how expensive it is to serve a SoTA model at 100 tokens a second. In addition to the $500k in capital costs, you also have significant electricity costs. This stuff is expensive because supply is much lower than demand. If everyone was to run their own hardware with a batch size of 1, we'd have 100x more demand for inference hardware and electricity than we do now, and…

But that's why you shouldn't expect local models to provide quick real-time answers, at least not with the same smarts as SOTA models running in the cloud. Slow batched inference (if possible - RAM capacity can obviously be a challenge with typical models and end-user hardware) can be a lot more effective.

Re: Local AI needs to be the norm

#652

Post by alfiedotwtf »

zozbot234 wrote: post #465
alfiedotwtf wrote: post #427

Earlier quoted context omitted.

What is everyone running DeepSeek v4 Flash with?! It’s currently unsupported on Llama.cpp and vllm doesn’t support GPU+CPU MoE, so unless all of you have an array of DGX Sparks in your bedroom, what’s the secret sauce?!

https://www.github.com/antirez/ds4 (from Antirez of Redis fame) runs a 2-bit quant on Apple Silicon hardware and 96GB or 128GB RAM.

I've been keeping an eye on Antirez's Metal fork for llama.cpp, but I totally missed this. Whoa, nice. Giving it a go, thanks!!

Re: Local AI needs to be the norm

#653

Post by alfiedotwtf »

thot_experiment wrote: post #446
alfiedotwtf wrote: post #412

Earlier quoted context omitted.

I’m guessing Qwen3.6 for agentic coding and Gemma4 for non-coding stuff?

No, exactly the opposite actually. Qwen3.6 is too imprecise for long running agentic tasks. It doesn't have the same ability to check itself as Gemma does in my testing. I keep Qwen MoE in vram by default because there are tons of tasks i trust it to oneshot and it's 90tok/sec is unparalleled, anything where I don't want to have to intervene too much it can't be trusted.

Oh interesting. I've read that Gemma 4 is really good for creative stuff, but I'm mostly interested in agentic coding. Unfortunately, each time I use Gemma 4, I just get it stuck in loops.

Re: Local AI needs to be the norm

#654

Post by nsvd2 »

reisse wrote: post #212
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

I run Gemma locally on a 3090, it's amazing how useful it is to be able to call out to ollama in a bash script or cron job.

Re: Local AI needs to be the norm

#655

Post by fleventynine »

zozbot234 wrote: post #651
fleventynine wrote: post #647

Earlier quoted context omitted.

I think folks in this thread are underestimating how expensive it is to serve a SoTA model at 100 tokens a second. In addition to the $500k in capital costs, you also have significant electricity costs. This stuff is expensive because supply is much lower than demand. If everyone was to run their own hardware with a batch size of 1, we'd have 100x more demand for inference hardware and electricity than we do now, and…

But that's why you shouldn't expect local models to provide quick real-time answers, at least not with the same smarts as SOTA models running in the cloud. Slow batched inference (if possible - RAM capacity can obviously be a challenge with typical models and end-user hardware) can be a lot more effective.

My point is that it is WAY more efficient if we put the world's DRAM supply into a shared inference pool instead of stranding it in local machines where it won't have as high of batch size or utilization.

The cost of not being efficient is even higher DRAM costs than we have now, given supply and demand.

Re: Local AI needs to be the norm

#656

Post by goatlover »

voncheese wrote: post #619
DrewADesign wrote: post #603

Earlier quoted context omitted.

And this is exactly what the LLM provider industry is fighting tooth-and-nail. It’s not only because it doesn’t directly contribute to their bottom line, it also directly opposes the idea that LLMs are going to replace entire workers rather than enhance the abilities of individual workers. What we’re headed towards would have been a killer product and probably still shifted a bunch of capital to the bazillionaires ha…

As long as Apple and Google put reasonable AI capabilities on device, then software engineers will use those capabilities when it makes sense (the article gives lots of good examples of capabilities that make sense to run locally). As the author notes, it's cheaper and more reliable to run these things locally. That also doesn't preclude LLM services from being massively successful, they'll just have to justify the p…

In an ideal world they will. In reality most will use online AI, because it's path of least resistance and more familiar.

Re: Local AI needs to be the norm

#657

Post by goatlover »

virgil_disgr4ce wrote: post #645
fridder wrote: post #635

Earlier quoted context omitted.

at the moment yes. The one possible silver lining with all of the current hardware crunch is that it _should_ force some hardware advancements. The last couple years hardware has been kinda boring. My m1max is still zippy as all hell and doesn't really need to be upgraded, unless I am committing to local AI inference.

> it _should_ force some hardware advancements I'm very curious what kind of hardware advancements you're imagining. Because we're already kind of near a physical wall regarding heat dissipation on phones. I mean hey, maybe foundational physics will surprise the world with a radical breakthrough that disappears heat into a black hole or something, but I sure wouldn't hold my breath

Launch everyone's phones into space.

Re: Local AI needs to be the norm

#658

Post by throwaway5752 »

virgil_disgr4ce wrote: post #645
fridder wrote: post #635

Earlier quoted context omitted.

at the moment yes. The one possible silver lining with all of the current hardware crunch is that it _should_ force some hardware advancements. The last couple years hardware has been kinda boring. My m1max is still zippy as all hell and doesn't really need to be upgraded, unless I am committing to local AI inference.

> it _should_ force some hardware advancements I'm very curious what kind of hardware advancements you're imagining. Because we're already kind of near a physical wall regarding heat dissipation on phones. I mean hey, maybe foundational physics will surprise the world with a radical breakthrough that disappears heat into a black hole or something, but I sure wouldn't hold my breath

eGPU cradles, presumably, for people with intense local model execution requirements until it can be made to work in the device? This is exactly like the POS dongles Square had until tap to pay was more widespread?

Re: Local AI needs to be the norm

#659

Post by nsvd2 »

katzito wrote: post #644

Most people are lazy (which is (mostly) good) and don't care (which is (mostly) not good), as Gmail has proven since 2004 (according to Google AI). Still waiting for those analog AI chips that were supposed to make it lightning fast using minimal energy...

Assuming you're talking about Taalas, they have a live demo for inference on their HC1 chip.

Re: Local AI needs to be the norm

#660

Post by fsflover »

dTal wrote: post #640
leoc wrote: post #629

(I am not an expert on anything.) One happy circumstance here is that while the RAM cartel is chasing Big AI's money today, in the medium term its self-interest probably makes it a supporter of local AI. A new, compelling reason to have 128GiB, 256GiB or more of VRAM on all your devices? You can be sure that the dollar signs are glowing in their eyes already. The less efficient use of VRAM by personal devies (any giv…

Just because we'll have to pay for the hardware, doesn't mean we'll have meaningful control. Look at what happened with phones - weak and limited slaves to the mothership, secured against pesky users with powerful encryption, yet costing more than a vastly superior laptop; quasi-mandatory platforms for highly addictive experiences, centered around the flow of information. And now with LLMs we can create even more fab…

> Look at what happened with phones - weak and limited slaves to the mothership, secured against pesky users with powerful encryption

Not all phones are like this. GNU/Linux phones obeying users exist too.

Post reply on HN