Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

411–420 of 804 posts

Re: Local AI needs to be the norm

#411

Post by intothemild »

lelanthran wrote: post #166
krupan wrote: post #153

Earlier quoted context omitted.

That's not what this thread is about? We're saying some new breakthrough is needed, someone said it already has happened, and I'm asking if it really has. Has it? I don't think so, those models are not in some way fundamentally different than other LLMs

> We're saying some new breakthrough is needed, someone said it already has happened, and I'm asking if it really has. I didn't read "and how were those models trained" as "Are we there yet?"

There's a percentage of people who love to question how the open models were trained.. they are almost always going to try and make some argument about using the closed frontier models for distillation as some form of theft.

Just totally forgetting that the frontier models themselves stole an insane amount to get to where they are.

It's theft all the way across the board, and when someone tries to make the argument that open models theft is bad, but Altman or Amodei's theft is good.. they are revealing a lot about themselves

Re: Local AI needs to be the norm

#412

Post by alfiedotwtf »

thot_experiment wrote: post #403
root_axis wrote: post #290

Earlier quoted context omitted.

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model's reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.

Flat wrong. Q6 Gemma 31b feels a lot like opus 4.5 to me when run in a harness so it can retrieve information and ground itself. The gap is not that big for a lot of usecases. Qwen MoE is fast as fuck locally for things that are oneshottable. I have subscriptions to all the major providers right now and since Gemma 4 and Qwen 3.6 came out I haven't hit limits a single time. I'm actually super surprised by the number…

I’m guessing Qwen3.6 for agentic coding and Gemma4 for non-coding stuff?

Re: Local AI needs to be the norm

#413

Post by hydra-f »

Unless there's a breakthrough or a transition to diffusion models, it's hard to imagine them becoming an affordable commodity

Small models are still in their infancy, and there's still much to sort out about and around them, as well

Re: Local AI needs to be the norm

#414

Post by aaronblohowiak »

root_axis wrote: post #308
wincy wrote: post #294

Earlier quoted context omitted.

Won’t these H100s drop in price in a few years? With the data center build out surely these will become 1/10th the price and you’ll be able to set up a local LLM as good as opus 4.7. Even if the frontier model become more advanced, and memory hungry, you could use the same power usage as your oven to run a current day frontier model as needed? If I could drop $10,000 to have an effectively permanent opus 4.7 subscrip…

> Won’t these H100s drop in price in a few years Doubtful. The increase in demand is greatly outpacing supply, and all signs point to a continued acceleration in demand > If I could drop $10,000 to have an effectively permanent opus 4.7 subscription today, I would. lol well obviously, but realistically that price point is going to be closer to $100k, with a perpetual $1k a month in power costs.

taalas!!!

Re: Local AI needs to be the norm

#415

Post by cbg0 »

thot_experiment wrote: post #403
root_axis wrote: post #290

Earlier quoted context omitted.

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model's reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.

Flat wrong. Q6 Gemma 31b feels a lot like opus 4.5 to me when run in a harness so it can retrieve information and ground itself. The gap is not that big for a lot of usecases. Qwen MoE is fast as fuck locally for things that are oneshottable. I have subscriptions to all the major providers right now and since Gemma 4 and Qwen 3.6 came out I haven't hit limits a single time. I'm actually super surprised by the number…

Your n=1 might not be very relevant outside your personal use. In less contaminated benchmarks Gemma 4 is way below Sonnet 4.5, let alone Opus models: https://swe-rebench.com/

Re: Local AI needs to be the norm

#416

Post by agnishom »

adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

The article is not about those use cases. There are plenty of use cases for which local models are already pretty good

Re: Local AI needs to be the norm

#417

Post by driese »

emadb wrote: post #393
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

Do you think small models will arrive? I mean if I need to write a web application in typescript why should I use a model that knows all the programming languages and it is able to reply to any questions about almost everything? I just a need a small performant model that knows how to write web applications in typescript. That could be very helpful and easy to run on my laptop.

For the same reason that a human who is fluent in five languages can probably express themselves better in either one compared to human that only speaks one, while also having a more nuanced understanding of general grammar. From what I know, learning on a more diverse set makes a model better overall.

Re: Local AI needs to be the norm

#418

Post by anemoknee »

tzm wrote: post #408

People want local AI, but only if UX is good. Tooling/harness quality may matter as much as model quality. I think the future will probably be a hybrid of: 1. local AI for simple, private, everyday tasks 2. online AI for very hard or long tasks

The Clippy app someone made and posted here a while back is the perfect average person LLM interface;

https://felixrieseberg.github.io/clippy/

Re: Local AI needs to be the norm

#420

Post by t_mahmood »

SilentM68 wrote: post #377
winocm wrote: post #247

Earlier quoted context omitted.

Perhaps I am the odd one out here, but a small part of me wants to see what happens when you run a proprietary SOTA model on a laptop.

Not sure if this is exactly the scenario you envision but I run ComfyUI on an Acer Helio 300 laptop, from four years ago. Has 16GB RAM, NVIDIA GeForce RTX 2060 w/6144MiB of VRAM and have generated a few images using "NetaYumev35_pretrained_all_in_one.safetensors" @ 10.6GB checkpoint, (well beyond the 6GB capacity of the RTX 2060 card). That being said, it takes more than 10 minutes to complete the task. Of course, I…

I'm not running local for exactly the same reason, to not stress my components. As it seems we are in for a long haul due to this AI bubble (can't wait for it to pop) so need to make sure I survive this madness, as for sure I can't afford to replace anything right now.
Post reply on HN