Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

681–690 of 804 posts

Re: Local AI needs to be the norm

#681

Post by zozbot234 »

jimbokun wrote: post #679
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

Has anyone tried to calculate the break even cost of buying a PC to run an LLM locally, vs the amount of tokens you could get from an AI provider?

The basic answer: very much not worth it at face value, becomes arguably worth it once you start worrying about future rug pulls from the big AI providers. (And that does include the market for third-party inference, at least at present.) It's also worth it if you have existing hardware to repurpose, but that's obvious and not what you were asking about.

Re: Local AI needs to be the norm

#682

Post by runfreeapps »

Any project that requires a local model should always be the way to go on first attempts and if the functionality is acceptable should stay with local models. Token burn is a serious problem and will ultimately lead developers to ask one question "Do I really need Opus xyz?" For most requirements of standard applications the answer is no. So using open-source llm models that are integrating in practical use-cases to create a value-add not for 'hey look I have AI in my app, sign up please.' Open source models are competing well and is the way to go for the majority of projects and mindsets do have to change and I see them changing this way rapidly. You don't have to host your open-source llm locally but host it with a 3rd party, it is cost-effective and the token burn is not a barrier.

Re: Local AI needs to be the norm

#683

Post by munk-a »

andybak wrote: post #591
Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

> draw a pretty good pelican on a bike. You mean the famously hard task? The one picked because it stretches frontier models to their limits?

It was a famously hard task. It was an ingenious idea for an unexpected task that falls outside of the bounds of predictable normal input but is still readily comprehended by the public.

Unfortunately, as soon as it's a famously hard task trainers know they need to succeed at it and it loses a lot of the power to detect correctness.

Re: Local AI needs to be the norm

#684

Post by willchis »

csheaff wrote: post #650
virgil_disgr4ce wrote: post #645

Earlier quoted context omitted.

> it _should_ force some hardware advancements I'm very curious what kind of hardware advancements you're imagining. Because we're already kind of near a physical wall regarding heat dissipation on phones. I mean hey, maybe foundational physics will surprise the world with a radical breakthrough that disappears heat into a black hole or something, but I sure wouldn't hold my breath

water cooled pant pockets

The iPant? Or a Samsung WCPP1?

Re: Local AI needs to be the norm

#685

Post by alfiedotwtf »

zozbot234 wrote: post #668
alfiedotwtf wrote: post #652

Earlier quoted context omitted.

I've been keeping an eye on Antirez's Metal fork for llama.cpp, but I totally missed this. Whoa, nice. Giving it a go, thanks!!

What kind of hardware are you planning to run this on? As mentioned already, I've been trying to understand how gracefully it might degrade on 64GB RAM or perhaps lower (the total weights size is 80GB at the provided quant) using SSD offload for the weights, and then (assuming it works and doesn't just OOM) whether the tok/s figures might meaningfully improve in that scenario by running multiple sessions in parallel.

I've got a 4060 Ti 12Gb with 128Gb RAM. I was hoping once I could demonstrate to myself that I could run Deepseek v4 Flash locally (even at really slow speeds), then it would be worth my time and money to get something to run it > 20t/s.

... currently testing out Stepfun 3.5 Flash Q4_k_m as a stop gap (unless it blows my socks off first).

Re: Local AI needs to be the norm

#686

Post by alfiedotwtf »

doctorpangloss wrote: post #665
alfiedotwtf wrote: post #427

Earlier quoted context omitted.

What is everyone running DeepSeek v4 Flash with?! It’s currently unsupported on Llama.cpp and vllm doesn’t support GPU+CPU MoE, so unless all of you have an array of DGX Sparks in your bedroom, what’s the secret sauce?!

you can run it today with mlx if you have 256g or 512g mac studio. no "antirez" fork needed. it isn't that large of a model and the compressed kv implementation is not that complicated the problem is that they released the model in a quantized format that is more complex than it appears, and people make a lot of mistakes working with it. it is quantization-aware-trained, so you can't "just" upscale it and scale down.…

Unfortunately I didn't get a Mac with big ram at the time it was cheap, and I'd personally focus on moving away from Apple and going Linux fulltime at work and home (currently Macbook for laptop connected to my big rig, well it's not that big compared to the AI people in here).

Re: Local AI needs to be the norm

#687

Post by platevoltage »

hparadiz wrote: post #382
platevoltage wrote: post #362

Earlier quoted context omitted.

You're going to see major cope once that bargain $200/month plan goes away, and every person or company that has embedded these services into their workflows gets to see their actual costs.

Have you actually tried this stuff or are you just saying stuff you hear on the internet?

Yes. I have tried this stuff. I really don't see how my use, or non-use of AI APIs changes this reality. Github Copilot announced it's going to per-token pricing in less than a month. I heard that on the internet by the way.

Re: Local AI needs to be the norm

#688

Post by geon »

virgil_disgr4ce wrote: post #645
fridder wrote: post #635

Earlier quoted context omitted.

at the moment yes. The one possible silver lining with all of the current hardware crunch is that it _should_ force some hardware advancements. The last couple years hardware has been kinda boring. My m1max is still zippy as all hell and doesn't really need to be upgraded, unless I am committing to local AI inference.

> it _should_ force some hardware advancements I'm very curious what kind of hardware advancements you're imagining. Because we're already kind of near a physical wall regarding heat dissipation on phones. I mean hey, maybe foundational physics will surprise the world with a radical breakthrough that disappears heat into a black hole or something, but I sure wouldn't hold my breath

More likely it would force software advancements. Current models are horribly inefficient.

Re: Local AI needs to be the norm

#689

Post by pianopatrick »

stuaxo wrote: post #484
pianopatrick wrote: post #384

Earlier quoted context omitted.

Currently I'm testing something like this just to see what happens. I have an old laptop with 4GB of RAM. I attached a USB drive with Gemma 4 31B model (which is 32.6 GB). Currently the laptop is running llama.cpp and trying to respond to a prompt by streaming the model from disk. The USB drive light is flickering, showing something is happening. It's been about 8 hours since I entered the prompt and I've gotten abou…

Nice. What did you use to do this, something standard like llamacpp or something else like vllm or your own contraption ?

llama.cpp

It's now spit out about 40 tokens after maybe 18 hours and has not finished the "thinking" stage of responding to the prompt. I'll let it keep running to see what happens

Re: Local AI needs to be the norm

#690

Post by yieldcrv »

nullsanity wrote: post #337
yieldcrv wrote: post #312

Earlier quoted context omitted.

I need to see these proper harnesses I tried oMLX and OpenCode a few weeks ago and the 65k context window was useless, it tried to analyze a very small codebase before going full on agentic and ran out of context window immediately I don't have time to tweak 1,000 permutations of settings just re-prove that its not as smart as Opus 4.6 I need out the box multimodal behavior as similar as typing claude in the command…

Hey man, you can just say "I'm lazy, so I'm staying with the cloud. if I wanted to use my brain, I wouldn't be using AI, gosh" - it's much shorter.

all the money and clout is in considering people’s reported problems as valid and solving them

so when I encounter a common but invalidated friction, I explain it like I’m 5, understanding that many of the engineering and entrepreneurial problem solvers have the emotional intelligence of a 5 year old

Post reply on HN