Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

421–430 of 804 posts

Re: Local AI needs to be the norm

#421

Post by root_axis »

thot_experiment wrote: post #403
root_axis wrote: post #290

Earlier quoted context omitted.

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model's reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.

Flat wrong. Q6 Gemma 31b feels a lot like opus 4.5 to me when run in a harness so it can retrieve information and ground itself. The gap is not that big for a lot of usecases. Qwen MoE is fast as fuck locally for things that are oneshottable. I have subscriptions to all the major providers right now and since Gemma 4 and Qwen 3.6 came out I haven't hit limits a single time. I'm actually super surprised by the number…

Sorry but you're just seeing what you want to see. The idea that a 31b model is anywhere even in the ballpark of something like Opus 4.5 is just absurd on its face.

Re: Local AI needs to be the norm

#422

Post by h05sz487b »

I really want this to be true. For me getting all models to run to the best of my hardwares ability and the cli tool to also make best use of the model is still a headache. I had coding models not being able to do a search and replace depending on the tool through which they were called, visible elements in my message flow, agents doing a task, failing at the linter, then reverting everything again so the linter is happy and presenting the result as a "good compromise".

Right now it feels like we have all the pieces but nobody integrating all that into an amazing experience.

Re: Local AI needs to be the norm

#424

Post by dyauspitr »

root_axis wrote: post #308
wincy wrote: post #294

Earlier quoted context omitted.

Won’t these H100s drop in price in a few years? With the data center build out surely these will become 1/10th the price and you’ll be able to set up a local LLM as good as opus 4.7. Even if the frontier model become more advanced, and memory hungry, you could use the same power usage as your oven to run a current day frontier model as needed? If I could drop $10,000 to have an effectively permanent opus 4.7 subscrip…

> Won’t these H100s drop in price in a few years Doubtful. The increase in demand is greatly outpacing supply, and all signs point to a continued acceleration in demand > If I could drop $10,000 to have an effectively permanent opus 4.7 subscription today, I would. lol well obviously, but realistically that price point is going to be closer to $100k, with a perpetual $1k a month in power costs.

Why? These models are going to keep drastically improving and given all the new data centers token prices will probably drop a lot in the future. Seems shortsighted given the absurd timelines these things have been improving on.

Re: Local AI needs to be the norm

#425

Post by ninjahawk1 »

In my opinion, this is similar to the earlier internet and computers. Few households or individuals had access to state of the art computers, it was primarily research or more well-off individuals. Most random people didn’t really know what it was and certainly didn’t use one.

Now today, AI is very expensive and not readily accessible to most people without paying a good amount.

The early internet became now you can just get a free phone from phone companies so long as you get their extras. Then you get a ton of subscriptions and ad-ons, but you don’t have to spend money, could just use youtube with ads etc.

Local AI would similarly shift this dynamic to paying for access to plug-in’s and tools for your local AI to be able to use. Like how the subscription model works right now.

With local model advancements, such as specifically Qwen 3.6 35B A3B, this future is becoming more likely by the year IMO.

Re: Local AI needs to be the norm

#426

Post by swiftcoder »

ac29 wrote: post #208
nullc wrote: post #114

Earlier quoted context omitted.

> two 4090s is not consumer grade I think that is a very narrow perspective. Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is "not consumer"? I agree with your view that cheap tokens on SOTA are a trap-- people should use local AI or no AI.

> Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is "not consumer"? $50k is a median priced car in the US. I'd guess >99.9% of people do not own $4000 of GPUs. I consider myself a computer person and I dont think I even own $4000 of computer hardware in total

> I consider myself a computer person and I dont think I even own $4000 of computer hardware in total

A top-spec MacBook Pro is >$4k, so I assure you that plenty of computer people do own $4k of computer hardware.

Hell, most tech folks are wandering around with a ~$1k smartphone in their pocket too.

Re: Local AI needs to be the norm

#427

Post by alfiedotwtf »

doctorpangloss wrote: post #329
zozbot234 wrote: post #102

Earlier quoted context omitted.

No one runs SOTA models 24/7 for individual use or even for a single household or small business, whereas you can run your own hardware basically 24/7 for AI inference. With the new DeepSeek V4 series and its uniquely memory-light KV cache you can even extend this to parallel inference in order to hide memory bandwidth bottlenecks and increase compute intensity. This is perhaps not so useful on a 128GB or 96GB RAM Ap…

deepseek v4 flash on mlx at 1m context runs at 20 t/s decode on a mac studio m3 ultra with 512gb of RAM

What is everyone running DeepSeek v4 Flash with?!

It’s currently unsupported on Llama.cpp and vllm doesn’t support GPU+CPU MoE, so unless all of you have an array of DGX Sparks in your bedroom, what’s the secret sauce?!

Re: Local AI needs to be the norm

#429

Post by testfrequency »

Local AI is definitely going to be the future as these models continue to advance at the rapid pace they already are.

This is why I believe OAI and Anthropic I’ve been so aggressive at offering services outside of their pure models like Claude Design. This is what will be competitive and keeping people subscribed.

Re: Local AI needs to be the norm

#430

Post by alfiedotwtf »

reissbaker wrote: post #187
CamperBob2 wrote: post #157

Earlier quoted context omitted.

It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. Not if you're OK with 4-bit quantization. More like $30K-$50K one time. Spring for 8 RTX6000s instead of 4, and you can use the full-precision K2.6 weights ( https://github.com/local-inference-lab/rtx6kpro/blob/master/... ).

RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waiting before any tokens come out), but it's true it technically works. I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usa…

If 8 x RTX 6000 is getting you 20s before initial token, how are cloud vendors doing this?
Post reply on HN