Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

741–750 of 804 posts

Re: Local AI needs to be the norm

#741

Post by alfiedotwtf »

zozbot234 wrote: post #707
alfiedotwtf wrote: post #686

Earlier quoted context omitted.

Unfortunately I didn't get a Mac with big ram at the time it was cheap, and I'd personally focus on moving away from Apple and going Linux fulltime at work and home (currently Macbook for laptop connected to my big rig, well it's not that big compared to the AI people in here).

What kind of RAM does your MacBook have? It might still be worth experimenting w/ DS4 using disk offload, though it would be dog slow at best and the RAM would be much too limited for meaningful parallelism, especially for larger contexts.

This might be my only hope until RAM prices come down to human levels again

Re: Local AI needs to be the norm

#742

Post by platevoltage »

hparadiz wrote: post #713
platevoltage wrote: post #712

Earlier quoted context omitted.

Fantastic. I run local models too. This was specifically about APIs

Why do you think costs will go up when the competition is increasing while hardware prices per compute cycle go down? That's the part I don't get.

Going from a fixed monthly fee to a per token billing structure is not a cost reduction, and we are watching that happen in real time.

Re: Local AI needs to be the norm

#743

Post by selectedambient »

Agree. We ought to be measuring the minimum viability of lesser parameter, local models for specific tasks. You don't need opus 4.7 or sonnet 4.6 to accomplish some of these basic, yet tedious tasks, i.e. the news aggregator you demonstrated. Thinking about things like, how many parameters does it take to manipulate a pdf in every way possible with accurate results? Likely, a reason there isn't a coordinated push toward people running local models is the fact that your data couldn't be mined, manipulated, and abused; obviously outside pure capability of some of the frontier models (which truthfully some of which aren't even very good). While I think we may see more things like Apple's models, like you mentioned being run locally, I think we all know at the end of the day they're phoning home in some way (which if that is fine for you, fine). Again though, and you touch on this in the article, highly specified tasks that have a certain amount of redundancy built in are very suitable for these local models right now, without relying on enormous weights and token usage.

I have been working on a VERY SMALL local-first ai lab myself. nothing crazy, a text editor, a claw, and some lightweight models I started playing with. Absolutely looking for contributions as well.

Re: Local AI needs to be the norm

#744

Post by selectedambient »

selectedambient wrote: post #743

Agree. We ought to be measuring the minimum viability of lesser parameter, local models for specific tasks. You don't need opus 4.7 or sonnet 4.6 to accomplish some of these basic, yet tedious tasks, i.e. the news aggregator you demonstrated. Thinking about things like, how many parameters does it take to manipulate a pdf in every way possible with accurate results? Likely, a reason there isn't a coordinated push tow…

didn't want to lead with it but if interested: https://mithraeums.github.io/

Re: Local AI needs to be the norm

#745

Post by andy_ppp »

rusk wrote: post #543
andy_ppp wrote: post #375

Earlier quoted context omitted.

Fine tuning these models (at least with PPO or equivalent) requires even more VRAM than inference does, potentially 2-3 times more.

You could use PEFT? Operating on only a subset of weights is fairly standard practice nowadays …

Yes I used LoRA and it’s fine but I’m not convinced the model doesn’t end up more stupid and less general

Re: Local AI needs to be the norm

#746

Post by DrewADesign »

voncheese wrote: post #732
DrewADesign wrote: post #664

Earlier quoted context omitted.

> That also doesn't preclude LLM services from being massively successful, they'll just have to justify the pricing and complexity that comes with their adoption, just like any other product. What is completely different from every other product is how much they’re spending, and how much they’re obligating themselves to spend going forward. I think there’s a very good chance that the existing providers could be miles…

Yeah that's a good callout for sure, the spending here is nuts so agree that it's not "just another business that has to price itself right to be competitive". I guess if the time horizons is long, like 20 years, then maybe the spending, as it begins to amortize, gets more in line? I was thinking that a comparison could be to cloud providers, each of which had to spend a lot of money to build out datacenter before ma…

The only way that this even vaguely works, best I can tell, would be on that decade-or-two timeline, but therein lies the problem: all this money getting pumped into data centers right now is going to produce data centers that are running old, inefficient, slow GPUs by 5-years-from-now standards. And GPUs are by far the most expensive part of these data centers… having the buildings is barely an asset. We’re investing all the money in right now’s technology in one of the fastest moving hardware segments and for some inexplicable reason, think that will lead to a sustainable advantage. What’s to stop someone 5 years from now, waiting for the dust to settle, then spending way less money for more compute and just mopping the floor with everybody in this sector… and that’s (unreasonably, IMO) assuming that local applications won’t become good enough to take too large a bite from their business before that.

And look at the difference in spending between their building out general-purpose-computing cloud data centers that even then, had potential use cases if the business failed. What are they going to do… start a massive, extremely expensive pre-rendered online gaming service? Only render Disney movies?

I dunno. None of this makes sense to me.

Re: Local AI needs to be the norm

#747

Post by zozbot234 »

DrewADesign wrote: post #746
voncheese wrote: post #732

Earlier quoted context omitted.

Yeah that's a good callout for sure, the spending here is nuts so agree that it's not "just another business that has to price itself right to be competitive". I guess if the time horizons is long, like 20 years, then maybe the spending, as it begins to amortize, gets more in line? I was thinking that a comparison could be to cloud providers, each of which had to spend a lot of money to build out datacenter before ma…

The only way that this even vaguely works, best I can tell, would be on that decade-or-two timeline, but therein lies the problem: all this money getting pumped into data centers right now is going to produce data centers that are running old, inefficient, slow GPUs by 5-years-from-now standards. And GPUs are by far the most expensive part of these data centers… having the buildings is barely an asset. We’re investin…

These datacenters are already running old, inefficient, slow GPUs from five years ago in addition to newly released cards, because anything newer than that is extremely bottlenecked and they need all the compute they can get. Why should it be any different in five years' time? Even nVidia is rumored to be about to bring back the RTX 3060 which is an Ampere architecture card that got released around 2021. It's just fine.

Re: Local AI needs to be the norm

#748

Post by alfiedotwtf »

thot_experiment wrote: post #693
alfiedotwtf wrote: post #653

Earlier quoted context omitted.

Oh interesting. I've read that Gemma 4 is really good for creative stuff, but I'm mostly interested in agentic coding. Unfortunately, each time I use Gemma 4, I just get it stuck in loops.

This is probably a precision thing, I think there's a really big difference in long running tasks between q4 and q6.

Ok, you’ve given me the umph to try again. Thanks!

Re: Local AI needs to be the norm

#749

Post by stuaxo »

thot_experiment wrote: post #493
stuaxo wrote: post #487

Earlier quoted context omitted.

What harness are you using ? I'm going to switch to local LLMs for most stuff soon.

Overall using screentime as the metric, derived from some imperfect logging and vibes it's about 50% OpenCode 15% Continue 15% my homebrew bullshit 13% Claude Code and 7% Cline. I've been deep on agentic stuff lately (1.3wks aka 3 months of AI time), there are only so many hours in the day to duplicate work and AB test, but in the past I've sworn by Qwen Coder + llama.vim and I still enjoy that workflow for deep work…

Interesting.

I stopped doing local stuff for a bit when I realised I didn't know how well it is supposed to work so have been on Claude for a few months now.

I think I'll try OpenCode this time.

Usually I do stuff in devcontainers, qwen code (non local) was the only time I managed to lose some work as it got confused when I ran out of tokens.

There's still quite a way to go - it does seem like Claude code itself is pretty badly coded, so I think there is a space for open source to come in with a high quality harness at some point.

Re: Local AI needs to be the norm

#750

Post by CamperBob2 »

fleventynine wrote: post #672
zozbot234 wrote: post #663

Earlier quoted context omitted.

Much of the world's DRAM stock is sitting idle in consumers' local machines and on-prem servers. If that DRAM gets some use, even "inefficiently", that's a meaningful decrease in demand.

That DRAM would get even more use if it was removed from these machines and placed into a shared pool :) I joke, but thanks to the brutal DRAM market there has been some movement in this direction lately...

I think the question of who controls the model is far more pressing than the question of who owns the DRAM.

It's easy to rattle off a half-dozen different vectors of likely enshittification over the next few years -- ranging from increasing censorship, to lower rate limits, to removal of existing features and forced addition of unwelcome new ones, to extortionate price increases, to unexplained and irreversible account bans. The only way to avoid them all is by running weights you own on hardware you control.

How smart and how fast is your local model? Those are certainly important questions, but "Does it exist at all?" is more important.

Post reply on HN