Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

191–200 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#191

Post by swader999 »

pupdogg wrote: post #189
mattnewton wrote: post #50

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

You can run a setup of 8x 4090 GPUs using 4x 1200W 240V power supplies (preferably HP HSTNS-PD30 Platinum Series), with a collective use of just around 20-amps, meaning it can easily run on a single 240V 20-amp breaker. This should be easily doable in a home where you typically have a 100 to 200A main power panel. Running 4x 1200W power supplies 24 hours a day will consume 115.2 kWh per day. At an electricity rate of…

You'll have helicopters over your house.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#192

Post by pupdogg »

swader999 wrote: post #191
pupdogg wrote: post #189

Earlier quoted context omitted.

You can run a setup of 8x 4090 GPUs using 4x 1200W 240V power supplies (preferably HP HSTNS-PD30 Platinum Series), with a collective use of just around 20-amps, meaning it can easily run on a single 240V 20-amp breaker. This should be easily doable in a home where you typically have a 100 to 200A main power panel. Running 4x 1200W power supplies 24 hours a day will consume 115.2 kWh per day. At an electricity rate of…

You'll have helicopters over your house.

Believe it or not, a buddy of mine was dealing with crazy high power bills for three months and couldn’t figure out why. He tried everything to cut back, but nothing worked. Finally, he called an electrician, who found a hidden short in a 30-amp circuit that was constantly drawing power without tripping the breaker. After fixing the issue, his bills went back to normal—no helicopters were involved during the process!

Re: Serving AI from the Basement – 192GB of VRAM Setup

#193

Post by w-ll »

swader999 wrote: post #191
pupdogg wrote: post #189

Earlier quoted context omitted.

You can run a setup of 8x 4090 GPUs using 4x 1200W 240V power supplies (preferably HP HSTNS-PD30 Platinum Series), with a collective use of just around 20-amps, meaning it can easily run on a single 240V 20-amp breaker. This should be easily doable in a home where you typically have a 100 to 200A main power panel. Running 4x 1200W power supplies 24 hours a day will consume 115.2 kWh per day. At an electricity rate of…

You'll have helicopters over your house.

Good thing i can see them coming, and happy to offer anyone a cup of coffee/tea to understand im just the biggest nerd in the hoa.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#195

Post by killingtime74 »

Did everyone just miss the fact that the post says the intention is to run Llama 3 405b but it has less than 1/4 of the VRAM required to do so? Did you just change your goals mid build? It's commonly known how much ram is required for a certain parameter size.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#196

Post by fragmede »

tourmalinetaco wrote: post #185
tcdent wrote: post #70

Earlier quoted context omitted.

I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

I’m hardly surprised, this is primarily a programming discussion website, and the highest voltage read on an average day here is in mV. It’s natural to be leery of things you have no experience in.

your car is 12 volts, and USB is 5 volts; 12 or up to 20 these days for laptop charging. My computer's CPU is probably 1.8 volts but I can't remember the last time I had my multimeter on that, but that's still more than millivolts.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#197

Post by schaefer »

Amazing writeup. And what a heavy hitter of an inaugural blog entry...

This might be the right time to ask: So, on the one hand, this is what it takes to pack 192gb of Nvidia flavored vram into a home server.

I'm curious, is there any hope of doing any interesting work on a MacBook Pro Which currently can be max-spaced at 128 GB of unified memory (for the low, low price of $4.7k).

I know there's no hope of running cuda on the macbook, and I'm clearly out of my depth here. But the possibly naive day-dream of tossing a massive LLM into a backpack is alluring...

Re: Serving AI from the Basement – 192GB of VRAM Setup

#198

Post by fragmede »

beAbU wrote: post #137
lolinder wrote: post #116

Earlier quoted context omitted.

We have an electric kettle in the US and it runs just fine drawing 1500W. You're correct that the dryer is on a larger circuit, though.

> and it runs just fine drawing 1500W. You think that this is "just fine" because you've never experienced the glory that is a 3kW kettle!

Japanese water boilers keep it boiling so there's no wait at all!

Re: Serving AI from the Basement – 192GB of VRAM Setup

#199

Post by dymk »

rootusrootus wrote: post #156
beAbU wrote: post #137

Earlier quoted context omitted.

> and it runs just fine drawing 1500W. You think that this is "just fine" because you've never experienced the glory that is a 3kW kettle!

I just get 99C water from a tap next to my kitchen sink. Why do people still use kettles?

Because they don’t have a spare few grand for an instant hot plus installation

Re: Serving AI from the Basement – 192GB of VRAM Setup

#200

Post by defrost »

swader999 wrote: post #191
pupdogg wrote: post #189

Earlier quoted context omitted.

You can run a setup of 8x 4090 GPUs using 4x 1200W 240V power supplies (preferably HP HSTNS-PD30 Platinum Series), with a collective use of just around 20-amps, meaning it can easily run on a single 240V 20-amp breaker. This should be easily doable in a home where you typically have a 100 to 200A main power panel. Running 4x 1200W power supplies 24 hours a day will consume 115.2 kWh per day. At an electricity rate of…

You'll have helicopters over your house.

Already have to deal with Robinson R22's during mustering season, a few more won't hurt.
Post reply on HN