Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

221–230 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#221

Earlier quoted context omitted.

100% agree, anything beyond 4x gpu’s is getting into the very annoying to power territory and makes the cloud very attractive. I already can trip a 15A circuit on 115v power with just 3x4090s and a SPR-X cpu. It also costs a lot to power. In the summer, 2x more than you expect, because unless it’s outside, you need cool 1000+ watts of extra heat with your AC. All that together and runpod starts to look very tempting!

Getting that circuit installed was pretty cheap likely because its in an unfinished and unconditioned basement. The basement stays comfortable even during the summer. The heat does seem to work its way into the rest of the house but the additional cooling load is only about 20% more than usual. It lowers the heating cost about the same amount during the winter so it works itself out.

Yeah location of your place, climate, and placement of the server in the house will affect this a lot. I'm on the top story of a building, even in the winter I rarely need to turn on my heat, just getting by on the waste heat of the rest of the building. My assortment of machines will easily keep the living room at 25C+ with a window open unless it's below 10C out! If I could keep the servers in the cool parking garage, I'd save a lot of money...

Getting a circuit put is also much more difficult in a shared building...

Runpod has 3090s for .43 per hour! .22 spot. If your power costs .3$ per kWh, and you need to spend _another_ .3$ per kWh in cooling, say if you live in apartment in the Bay Area and it's summer, that's ~48 days to equal the cost of 30 days on runpod. So you are still saving some money, though much less than you might think and possibly spending more than spot instances!

Re: Serving AI from the Basement – 192GB of VRAM Setup

#222

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

Not speaking from direct experience building a rig like this but the blog post mentions having 3 power supplies so the most direct solution would be to put each on their own dedicated circuit. As long as you have space in your electrical box this is straightforward to do though I would recommend having an electrician do the wiring if you aren’t experienced with that type of home electrical work.

Actually, putting each PSU on its own circuit is crazy dangerous. In the scenario of your suggestion, if one goes out, you are in for a fire. Highly recommend against that.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#223

Earlier quoted context omitted.

Huh, what?! Mega TIL moment for me as an Australian with an electric kettle and dryer plugged into whatever power socket I wish! Reminds me of this great Technology Connections video: https://youtu.be/jMmUoZh3Hq4?si=3vSMHmU2ClwNRtow

I'm pretty sure in our current rental the kitchen and laundry are on the same circuit, which means I'll often have the dryer, washing machine, kettle, toaster and microwave drawing power at the same time. It's never been an issue.

What kind of a dryer? Because that cannot be right. Dryers require a 30amp 240volt dedicated breakers by code in most counties in most states nowadays.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#224

Earlier quoted context omitted.

I'm pretty sure in our current rental the kitchen and laundry are on the same circuit, which means I'll often have the dryer, washing machine, kettle, toaster and microwave drawing power at the same time. It's never been an issue.

Insane

If that is true and OP is not just confused, he should sue his landlord, and I am not even kidding!!!

Re: Serving AI from the Basement – 192GB of VRAM Setup

#225

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

Take your typical 'GPU node', which would be a Dell/HP/SuperMicro with 4-8 NVIDIA H100's and a single top high level AMD/Intel CPU. You would need 2-4 240v outlets (30A). In the real world you would plug them into a PDU such as: https://www.apc.com/us/en/product/AP9571A/rack-pdu-basic-1u-... Each GPU will take around 700W and then you have the rest of the system to power, so depending on CPU/RAM/storage... And then y…

Yeah, you get it. I have two PDUs with NEMA L6-30P plugin, and two NEMA L6-30R outlet. Each outlet is connected to it's dedicated 30A 240v circuit (which gives you a total of 60A, or 6000w).

Cooling is its own story... Man, figuring out this process was hella of a journey

Re: Serving AI from the Basement – 192GB of VRAM Setup

#226

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

Not the OP, but I hired an electrician to put a 30A 240V circuit with a NEMA L6-30R recepticle next to my electrical panel. It was 600 CAD. You can probably get it done cheaper. He had to disconnect another circuit and make a trip to the hardwate store because I told him to bring the wrong breaker.

Yup, that's exactly what I had installed.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#227
post #102

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

This is funny as a european, since we have many, many groups where we reguarly will run 2kW, and some, loads. Really no issue, but I guess lower voltage makes it a problem.

Most outlets rely on 120v 15amp breakers here in the US, only washer/dryer/kitchenware require the higher end 240v breakers

Re: Serving AI from the Basement – 192GB of VRAM Setup

#228

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

I’ve ran 3x L40S on a 1650W PSU on a normal 120V 20A circuit.

You've got a much stronger heart than me man

Re: Serving AI from the Basement – 192GB of VRAM Setup

#229

Earlier quoted context omitted.

I’m hardly surprised, this is primarily a programming discussion website, and the highest voltage read on an average day here is in mV. It’s natural to be leery of things you have no experience in.

your car is 12 volts, and USB is 5 volts; 12 or up to 20 these days for laptop charging. My computer's CPU is probably 1.8 volts but I can't remember the last time I had my multimeter on that, but that's still more than millivolts.

That's technically correct, but irrelevant: you cannot kill yourself with 12 or 20V any more than with 10mV. 120V or 230V is another story.

That being said, it's still very easy not to kill yourself with 120/230V: just shut down the power before touching anything.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#230

Earlier quoted context omitted.

Not speaking from direct experience building a rig like this but the blog post mentions having 3 power supplies so the most direct solution would be to put each on their own dedicated circuit. As long as you have space in your electrical box this is straightforward to do though I would recommend having an electrician do the wiring if you aren’t experienced with that type of home electrical work.

Actually, putting each PSU on its own circuit is crazy dangerous. In the scenario of your suggestion, if one goes out, you are in for a fire. Highly recommend against that.

This might be one of those reasons why I'm not an electritian, but is it dangerous.

Back when I worked at a high-availability data center, all of our servers had dual psus, plugged into seperate circuits.

The transformer in the PSUs should electrical isolate the mains voltage from the low voltage side, so you aren't going to cause a short across the two circuts.

The only risk I see is a cascade failure, where the increased load on the second circuit causes its breaker to trip.

Post reply on HN