Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

671–680 of 804 posts

Re: Local AI needs to be the norm

#671

Post by jimbokun »

henry_kang wrote: post #600
Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

Actually you can do way more things than that. We have optimized it to process 2TB of high def videos on a M5 MBP in under 24 hours, including everything such as speech understanding, face recog, LLM and VLM. Super fun.

If Steve Jobs was alive Apple would have already demoed this as a new line of Macs with open weight models pre-installed with hooks into all of their existing content creation software.

And he would have the audience believing all the demos were running through third party AI providers, until at the last moment explaining “actually all of that ran on device with no connection to any external services.”

Re: Local AI needs to be the norm

#672

Post by fleventynine »

zozbot234 wrote: post #663
fleventynine wrote: post #655

Earlier quoted context omitted.

My point is that it is WAY more efficient if we put the world's DRAM supply into a shared inference pool instead of stranding it in local machines where it won't have as high of batch size or utilization. The cost of not being efficient is even higher DRAM costs than we have now, given supply and demand.

Much of the world's DRAM stock is sitting idle in consumers' local machines and on-prem servers. If that DRAM gets some use, even "inefficiently", that's a meaningful decrease in demand.

That DRAM would get even more use if it was removed from these machines and placed into a shared pool :) I joke, but thanks to the brutal DRAM market there has been some movement in this direction lately...

Re: Local AI needs to be the norm

#673

Post by CamperBob2 »

otabdeveloper4 wrote: post #387
reissbaker wrote: post #187

Earlier quoted context omitted.

RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waiting before any tokens come out), but it's true it technically works. I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usa…

> higher param count models will remain smarter for a looong time They're not smarter, they just know more stuff. You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM. The "smarts" comes from post-training, especially around tool use.

You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM.

That's one of the biggest remaining head-scratchers in this whole business. You do need all that unrelated stuff to make a good coding model.

Nobody knows why you can't build a coding model by training on nothing but code, CS texts, specifications, and case studies, but so far it appears that you can't.

Re: Local AI needs to be the norm

#674

Post by zozbot234 »

doctorpangloss wrote: post #665
alfiedotwtf wrote: post #427

Earlier quoted context omitted.

What is everyone running DeepSeek v4 Flash with?! It’s currently unsupported on Llama.cpp and vllm doesn’t support GPU+CPU MoE, so unless all of you have an array of DGX Sparks in your bedroom, what’s the secret sauce?!

you can run it today with mlx if you have 256g or 512g mac studio. no "antirez" fork needed. it isn't that large of a model and the compressed kv implementation is not that complicated the problem is that they released the model in a quantized format that is more complex than it appears, and people make a lot of mistakes working with it. it is quantization-aware-trained, so you can't "just" upscale it and scale down.…

If you have a 256 GB or 512 GB Mac Studio, the real game is to run multiple sessions in parallel in order to make the best use of your limited memory bandwidth. You'd have plenty of excess RAM for that given how small the KV cache is even at max context.

Re: Local AI needs to be the norm

#675

Post by jimbokun »

goatlover wrote: post #657
virgil_disgr4ce wrote: post #645

Earlier quoted context omitted.

> it _should_ force some hardware advancements I'm very curious what kind of hardware advancements you're imagining. Because we're already kind of near a physical wall regarding heat dissipation on phones. I mean hey, maybe foundational physics will surprise the world with a radical breakthrough that disappears heat into a black hole or something, but I sure wouldn't hold my breath

Launch everyone's phones into space.

Heat dissipation is even harder in space

Re: Local AI needs to be the norm

#676

Post by array_key_first »

tuananh wrote: post #423
DeathArrow wrote: post #395

Earlier quoted context omitted.

The full model is, not the quantized versions.

yeah that goes without saying. how can openweight, quantized version beat SOTA :)

Well it depends on the task. For agentic coding, more is more, but for tasks that normal consumers use them for there really is a ceiling. OCR, text to speech, that type of thing doesn't really improve when going to a SOTA model, so you'd just be wasting your money. I think local LLMs have more value than software engineers give them credit for.

Re: Local AI needs to be the norm

#677

Post by ActorNightly »

root_axis wrote: post #290
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model's reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.

Yes and no.

The best analogy is the difference between having N senior level engineers working for you, versus having N entry level engineers.

With frontier cloud models, you can give a single invocation one task, and it can figure everything out.

With local models, you have to manage the inputs and outputs quite a bit more, but you can achieve similar results for tasks you set up harnesses for. They are not as a good at finding the right answer internally from their own weights, but they are very capable of ingesting context and reformatting text - for example, for debugging, local models can debug issues quite well if you give them the error and documentation for a particular feature you are trying to implement.

Re: Local AI needs to be the norm

#678

Post by CamperBob2 »

alfiedotwtf wrote: post #430
reissbaker wrote: post #187

Earlier quoted context omitted.

RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waiting before any tokens come out), but it's true it technically works. I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usa…

If 8 x RTX 6000 is getting you 20s before initial token, how are cloud vendors doing this?

RTX6000s are great but they are several times slower than a real datacenter-grade GPU. They still use DDR memory rather than HBM, for example.

Re: Local AI needs to be the norm

#679

Post by jimbokun »

reisse wrote: post #212
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

Has anyone tried to calculate the break even cost of buying a PC to run an LLM locally, vs the amount of tokens you could get from an AI provider?

Re: Local AI needs to be the norm

#680

Post by zozbot234 »

array_key_first wrote: post #670
dTal wrote: post #640

Earlier quoted context omitted.

Just because we'll have to pay for the hardware, doesn't mean we'll have meaningful control. Look at what happened with phones - weak and limited slaves to the mothership, secured against pesky users with powerful encryption, yet costing more than a vastly superior laptop; quasi-mandatory platforms for highly addictive experiences, centered around the flow of information. And now with LLMs we can create even more fab…

Phones are such a travesty because they're so incredibly overpowered. I think there's a lot of people out there where their iPhone has more compute than their laptop or desktop, but it can't do 1/10th the amount of stuff. What a waste!

They're actually underpowered because they can't sustain that full compute over time like a desktop can, or even a laptop to some extent. That's a key limitation for AI.
Post reply on HN