Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

461–470 of 651 posts

Re: Running local models is good now

#461

Post by cheschire »

pornel wrote: post #344

[meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at…

Haves and have nots.

We aren’t wealthy enough to have the hardware that would make this good.

The people who have the money to buy a spare maxed out Mac mini just don’t get it. I see lots of folks with RTX 6000’s in threads like these. Or any RTX card that ends in “90”.

Cloud AI is what allows the proles to participate in the broader AI conversation, but not these AI conversations.

Re: Running local models is good now

#462

Post by rapidfl »

starshadowx2 wrote: post #386
rmunn wrote: post #7

This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…

Earlier I was thinking it's maybe comparable to paying for Netflix vs torrenting and running Plex or something. For the majority of normal, mainstream users I feel like most would just pay for the thing that is already setup and ready for them. There'll still be all the more techy or determined types who will do it themselves, I just wonder what the percentages of both groups will be.

if we get to the stage where the AI hardware is a more of a commodity and usability becomes 10x simpler, then people may buy their own hardware and run local models.

Everybody owns a car, washer, TV, etc today. Maybe one could finance a server-box/trailer costing $20k, trade it in every 7 years for a newer model, etc. Many people are going to own a $20k Optimus.

Re: Running local models is good now

#463

Post by milesvp »

rapind wrote: post #226
saghm wrote: post #136

Earlier quoted context omitted.

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…

> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…

If you think your data isn’t being hoovered up I’d like to point out that every model is possible due to federal crimes committed to obtain the information they were trained on. Regardless of how much you are paying, your data is worth another petty civil infraction.

Re: Running local models is good now

#464

Post by fragmede »

rapidfl wrote: post #462
starshadowx2 wrote: post #386

Earlier quoted context omitted.

Earlier I was thinking it's maybe comparable to paying for Netflix vs torrenting and running Plex or something. For the majority of normal, mainstream users I feel like most would just pay for the thing that is already setup and ready for them. There'll still be all the more techy or determined types who will do it themselves, I just wonder what the percentages of both groups will be.

if we get to the stage where the AI hardware is a more of a commodity and usability becomes 10x simpler, then people may buy their own hardware and run local models. Everybody owns a car, washer, TV, etc today. Maybe one could finance a server-box/trailer costing $20k, trade it in every 7 years for a newer model, etc. Many people are going to own a $20k Optimus.

The car, TV, washer, and whatever humanoid robot finds product market fit physically need to be in my house, or close to it, in order for them to be useful to me. Thanks to the Internet, the data center doesn't need to be, like at all. Economy of scale says that renting a slice of time on the most expensive GPU supercomputer out there is going to be faster and also probably cheaper since I'd only be getting a slice while the server is serving multiple users.

Re: Running local models is good now

#465

Post by mcmoor »

moregrist wrote: post #382
richardwhiuk wrote: post #220

Earlier quoted context omitted.

There's no economic reason why running a model locally should be better than using a cloud hosted version.

“There is no reason anyone would want a computer in their home." - Ken Olson, Founder of Digital Equipment Corporation, in 1977

In hindsight this is getting truer, what with the push of dumb terminal for everyone

Re: Running local models is good now

#467

Post by tpurves »

I do think local models are huge pending market opportunity for Apple. An M5 Ultra Mac Studio (if that exists) could be decent local AI machine, though so expensive as to stay niche. But by the M6/M7 generations and a recovery in DRAM affordability, the future could be interesting moment for them to deliver a compelling local AI platform that 'just works'. But I do think that a mini-pc that is easy to configure, can be always plugged-in, always on, higher power envelope than a laptop, but not obnoxiously loud and hot, is the right form-factor

Re: Running local models is good now

#468

Post by BenRacicot »

tpurves wrote: post #467

I do think local models are huge pending market opportunity for Apple. An M5 Ultra Mac Studio (if that exists) could be decent local AI machine, though so expensive as to stay niche. But by the M6/M7 generations and a recovery in DRAM affordability, the future could be interesting moment for them to deliver a compelling local AI platform that 'just works'. But I do think that a mini-pc that is easy to configure, can…

Agreed, this is what caused me to build. This thesis exactly.

Re: Running local models is good now

#470

Post by what »

echelon wrote: post #362
zozbot234 wrote: post #340

Earlier quoted context omitted.

H200s and other enterprise datacenter GPUs are completely overkill in any realistic single- or few-users inference scenario. They're hugely unbalanced towards compute capacity which will go almost entirely unused (i.e. wasted) unless you're running huge batches on a continued basis. I've argued many times that local inference engines should support batched inference on a somewhat smaller scale for a variety of reason…

If you're doing professional work in coding or video, you can easily saturate a single H200. This is what RunPod-type services are for. For instance, ComfyUI is an abomination that can't do half of what Nano Banana and Seedance 2.0 can do. And you have to sit around and wait 10x longer for single results. I can rent an H200 for $3.50 an hour. That's INSANELY cheap. I do not understand this split between hosted APIs a…

> I’m a contrarian that says things that rile up the anti-AI folks

That’s hardly contrarian here, lol.

Post reply on HN