Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

231–240 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#231

Post by anonymousDan »

yunohn wrote: post #83
fragmede wrote: post #69

Earlier quoted context omitted.

folding@home predates Bitcoin by eight years. the concept isn't inherent to grifters

Folding at home does not use a blockchain, further proving non-grifters don’t need it. That was the point being discussed, not distributed computing as a concept.

I don't think you are being fair to the previous poster. As I read it they are simply pointing out that there is precedent for such decentralized contribution of compute resources. However folding at home doesn't allow to reward users for their contributions AFAIK. So maybe if a Blockchain based reward system could be layered on top of that it could increase participation. That's a big if I grant you but don't see how it is completely inconceivable that such a thing might be possible.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#232

Post by illiac786 »

wkat4242 wrote: post #87
gizmo686 wrote: post #79

Earlier quoted context omitted.

The cloud companies do not make the hardware, they buy it like the rest of us. They are just going to be almost the entirety of the market, so naturally the products will built and priced with that market in mind.

Yes and they get deep discounts which we don't. Can be 40% or more! Of course the vendor can't make a profit with such discounts so they inflate the RRP. But we do end up paying that.

That’s the main problem is a market owned by enterprise customers. Consumers don’t matter, there is zero interest is competing for them, they’re too little. The discounts is a killer for example, well have to buy from a reseller each time, who of course will pocket a good proportion of the discount because there won’t be many resellers that sell to consumers…

I have seen very large ent customers get 80% discount on hardware - it’s mind boggling that the vendor is not going bankrupt.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#233

Post by illiac786 »

I dream from a future where the „home server with heat recuperation“ appliance will be common enough I can get a worker to install it physically for me - I have little electrical skills and zero plumbing skills. And I also hope that by then power consumption will have gone down.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#234

Post by rspoerri »

throwthrowuknow wrote: post #71
rspoerri wrote: post #20

Earlier quoted context omitted.

I'm running 70B models (usually in q4 .. q5_k_m, but possible to q6) on my 96Gbyte Macbook Pro with M2-Max (12 cpu cores, 38 gpu cores). This also leaves me with plenty of ram for other purposes. I'm currently using reflection:70b_q4 which does a very good job in my opinion. It generates with 5.5 tokens/s for the response, which is just about my reading speed. edit: I usually dont run larger models (q6) because of th…

Not going to work for training from scratch which is what the author is doing.

192GByte of RAM are not enough to train 405B models. Reflection 70B requires 140GByte of RAM in fp16, 405 would need ~810Gbyte of RAM.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#235

Post by yunohn »

anonymousDan wrote: post #231
yunohn wrote: post #83

Earlier quoted context omitted.

Folding at home does not use a blockchain, further proving non-grifters don’t need it. That was the point being discussed, not distributed computing as a concept.

I don't think you are being fair to the previous poster. As I read it they are simply pointing out that there is precedent for such decentralized contribution of compute resources. However folding at home doesn't allow to reward users for their contributions AFAIK. So maybe if a Blockchain based reward system could be layered on top of that it could increase participation. That's a big if I grant you but don't see ho…

I think the word blockchain confuses people, including you and the previous poster. Maybe you could clarify your “layering” idea and how it would work for further discussion.

Folding at home can track user contributions and issue micro/payments as they see fit. Crucially, this does not need an immutable chain of truth to do.

Instead, if we added a blockchain, then we would require 2 sets of participants - those who run the useful simulations for science, and those who run the useless calculations for the blockchain. A complete waste of resources.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#236

Post by raxxorraxor »

bigiain wrote: post #170
gizmo686 wrote: post #94

Earlier quoted context omitted.

A good engineer knows the difference between safe and dangerous. Setting up an AI computer is safe. Maybe you trip a circut. Maybe you interfere with something else running on your hobby computer. But nothing bad can really happen. Residential electrical is dangerous. Maybe you electrocute yourself. Maybe you cause a fire 5 years down the line. Maybe you cause a fire for the next owner because you didn't know to prot…

> Also, if you are at the point where you need to add a circut for power, you might need to seriously consider cooling, which could potentially be another side quest. There should be an easy/reliable way to channel "waste heat" from something like this to your hot water system. Actually, 4 or 5 kW continuous is a lot more than most domestic hot water services need. So in my usual manner of overcomplicating simple ide…

Instant water heaters use up to and sometimes even more than 27kW. Of course boilers use less, but still...

These aren't power requirements that are insurmountable. They would get pricey though and I wish my rig for computing would use something around .1kW under load...

Using the heat from PCs would be nice. I guess most just use them as electrical heaters right now.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#237

Post by raxxorraxor »

fragmede wrote: post #196
tourmalinetaco wrote: post #185

Earlier quoted context omitted.

I’m hardly surprised, this is primarily a programming discussion website, and the highest voltage read on an average day here is in mV. It’s natural to be leery of things you have no experience in.

your car is 12 volts, and USB is 5 volts; 12 or up to 20 these days for laptop charging. My computer's CPU is probably 1.8 volts but I can't remember the last time I had my multimeter on that, but that's still more than millivolts.

Probably meant milliampere, specifically 1 milliampere. But yes, usually lightweight engineers are familiar with TTL and limit themselves to 5V. 12V+ is another arcane realm you don't want to touch.

Some old serial ports had 12V and a high max current. The DIY things you attached here were prone to kill your mainboard.

Voltage/current is either 0 or 1. Anything higher kills software developers instantly.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#238

Post by wkat4242 »

illiac786 wrote: post #232
wkat4242 wrote: post #87

Earlier quoted context omitted.

Yes and they get deep discounts which we don't. Can be 40% or more! Of course the vendor can't make a profit with such discounts so they inflate the RRP. But we do end up paying that.

That’s the main problem is a market owned by enterprise customers. Consumers don’t matter, there is zero interest is competing for them, they’re too little. The discounts is a killer for example, well have to buy from a reseller each time, who of course will pocket a good proportion of the discount because there won’t be many resellers that sell to consumers… I have seen very large ent customers get 80% discount on h…

> I have seen very large ent customers get 80% discount on hardware - it’s mind boggling that the vendor is not going bankrupt.

Yes exactly. When I see what we pay for stuff at work...

Obviously the vendors don't have 80+% margins. So what do they do? Inflate the RRP to compensate. So they can give a huge discount that sounds good on paper.

But this makes it unviable to buy for consumers that do have to pay RRP.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#239

Post by wkat4242 »

Dylan16807 wrote: post #146
wkat4242 wrote: post #80

Earlier quoted context omitted.

They are easy to find but extremely expensive. I used to pay below 200€ for a 14TB Seagate 8 years ago. That's now above 300. And the bigger ones are even more expensive. For me these prices are prohibitive. Just like the A100s are (though those are even more so of course). The problem is the common consumer relying on the cloud so these kind of products become niches and lose volume. Also, the cloud providers don't…

Well if I look at Amazon I see a couple models of external 14TB for $190, and a brand new Exos 16TB for $230. Not too bad. Though personally I get much cheaper used drives and put them in RAID for a NAS. And they do have better sales.

Most of the cheap drives here are refurbs with questional quality. And those Exoses here are much more expensive sadly, especially if you choose only legit vendors on Amazon.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#240

Post by sireat »

I was under the mistaken impression that you could not go beyond 2x3090 for reasonable inference speed.

My assumption was that going beyond 2 cards incurs significant bandwidth penalty when going from NVLink between 2x3090s to PCIe for communicating between the other 3090s.

What kind of T/s speeds are you getting with this type of 8x3090 setup?

Presumably then even crazier 16x4090 would be an option for someone with enough PCIe slots/risers/extenders.

Post reply on HN