Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

771–780 of 804 posts

Re: Local AI needs to be the norm

#771

Post by petra »

zozbot234 wrote: post #747
DrewADesign wrote: post #746

Earlier quoted context omitted.

The only way that this even vaguely works, best I can tell, would be on that decade-or-two timeline, but therein lies the problem: all this money getting pumped into data centers right now is going to produce data centers that are running old, inefficient, slow GPUs by 5-years-from-now standards. And GPUs are by far the most expensive part of these data centers… having the buildings is barely an asset. We’re investin…

These datacenters are already running old, inefficient, slow GPUs from five years ago in addition to newly released cards, because anything newer than that is extremely bottlenecked and they need all the compute they can get. Why should it be any different in five years' time? Even nVidia is rumored to be about to bring back the RTX 3060 which is an Ampere architecture card that got released around 2021. It's just fi…

Unless some disrutive technology comes along in 5 years time. And many are working on exactly that.

Re: Local AI needs to be the norm

#772

Post by ElectricalUnion »

datadrivenangel wrote: post #239
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

In my experience once you get to ~30 gigs of ram for a model like Gemma4, the rest of the 128g of memory is simply nice to have. The speed and costs are what make it tough though, because its slower and more expensive than the same model served on a big accelerator card, and is going to be worse than a frontier model.

You need the rest of the ram for the context. If you don't want to end up with a toy context or quantized lossy context, is pretty easy to end up having to spend up 50+GB just for the KV cache, per simutaneous inference slot.

Re: Local AI needs to be the norm

#774

Post by zozbot234 »

ElectricalUnion wrote: post #772
datadrivenangel wrote: post #239

Earlier quoted context omitted.

In my experience once you get to ~30 gigs of ram for a model like Gemma4, the rest of the 128g of memory is simply nice to have. The speed and costs are what make it tough though, because its slower and more expensive than the same model served on a big accelerator card, and is going to be worse than a frontier model.

You need the rest of the ram for the context. If you don't want to end up with a toy context or quantized lossy context, is pretty easy to end up having to spend up 50+GB just for the KV cache, per simutaneous inference slot.

[dead]

Re: Local AI needs to be the norm

#775

Post by jesterson »

pjerem wrote: post #459
Tepix wrote: post #455

I'm pretty sure that AI assistants will become widespread. I consider it to be very careless to entrust your emails, your chats, your calendar, your notes, your calls, your pictures, your contacts, your location history, your waking hours, your files, your TODO list, i.e. stuff including your health data to the for-profit AI companies. The temptation to earn money with your data is just too great, plus the risk of th…

> I consider it to be very careless to entrust your emails, your chats, your calendar, your notes, your calls, your pictures, your contacts, your location history, your waking hours, your files, your TODO list, i.e. stuff including your health data to the for-profit AI companies. Still, we all do it with Google. (I don't do it anymore but i did it for mostly two decades so I include myself)

> Still, we all do it with Google

We don't. And never did.

Re: Local AI needs to be the norm

#776

Post by aceazzameen »

thot_experiment wrote: post #696
diordiderot wrote: post #521

Earlier quoted context omitted.

Maybe reaching for an analogy would be helpful here. Thot_experiment is saying that his 2016 Toyota Prius is a great and reliable car for his daily commute and running errands. Whereas everyone is screeching about its capability gap with a Lockheed Martin F35 lightning.

Yeah, thanks, though I think local models are at least a Cessna, which while being nothing like an F-35 can fly.

Flying is fun. But shooting Cessnas out of the sky is more fun!

I'm kidding around. I run 31b models myself too and am perfectly happy with them.

Re: Local AI needs to be the norm

#778

Post by 306bobby »

SilentM68 wrote: post #704
t_mahmood wrote: post #420

Earlier quoted context omitted.

I'm not running local for exactly the same reason, to not stress my components. As it seems we are in for a long haul due to this AI bubble (can't wait for it to pop) so need to make sure I survive this madness, as for sure I can't afford to replace anything right now.

I don't know that any AI bubble will pop. AI can be used to accelerate therapies, cures, make scientific advancements. Add to that, quantum science technology which if successful, should accelerate things, depending on who's the one at the wheel. Problem is the gap between now and then (e.g. age abundance). It's going to be a difficult road for good number of the population until that day comes. I'm scouting potentia…

Both the bubble popping and it's legitimate use cases can exist at the same time.

For example, the www bubble popped, but the Internet didn't go away

Re: Local AI needs to be the norm

#779

Post by voncheese »

zozbot234 wrote: post #747
DrewADesign wrote: post #746

Earlier quoted context omitted.

The only way that this even vaguely works, best I can tell, would be on that decade-or-two timeline, but therein lies the problem: all this money getting pumped into data centers right now is going to produce data centers that are running old, inefficient, slow GPUs by 5-years-from-now standards. And GPUs are by far the most expensive part of these data centers… having the buildings is barely an asset. We’re investin…

These datacenters are already running old, inefficient, slow GPUs from five years ago in addition to newly released cards, because anything newer than that is extremely bottlenecked and they need all the compute they can get. Why should it be any different in five years' time? Even nVidia is rumored to be about to bring back the RTX 3060 which is an Ampere architecture card that got released around 2021. It's just fi…

Yeah, conceptually this isn't all that different from new VM SKUs coming out in clouds. The costs and rate of change for AI hardware may be higher, and perhaps enough higher to mess up the math, but conceptually its a model that has been proven to work.
Post reply on HN