Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

141–150 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#141

Post by NavinF »

Dylan16807 wrote: post #86
NavinF wrote: post #23

Earlier quoted context omitted.

You could power limit the 3090s to fit a standard 120V*20A = 2400W outlet if you really want to. The default power limit is 350W each so you'll only lose a little perf. Also most rooms have multiple circuits. Just connect half the GPUs to each outlet. I already do this with my desktop PC because it has 2 PSUs. Also most homes in the US have 30A*240V = 7200W dryer/stove outlets in the kitchen, laundry room, garage, et…

A standard outlet is 15A, and you're only allowed to use 80% continuously. So now you're dropping to 150W each. Using two circuits is viable.

In the bay area every home I've lived in had 20A circuits for everything.

The NEC 80% rule is only relevant in data centers where someone enforces it. The time-current curve for any 20A breaker shows that it will take infinite time to trip when you draw 20A. This is correct behavior because 12awg wire can handle 20A indefinitely

Re: Serving AI from the Basement – 192GB of VRAM Setup

#142

Post by lolinder »

blibble wrote: post #138
lolinder wrote: post #116

Earlier quoted context omitted.

We have an electric kettle in the US and it runs just fine drawing 1500W. You're correct that the dryer is on a larger circuit, though.

I get bored and tend to wander off waiting for it to boil at 3kW 1.5kW must be absolute agony

I mean... yes, I don't sit around waiting for the kettle to boil. But if I fill it and start it first the water is already boiling by the time I get everything out, so it's not like any time is wasted as is.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#143

Post by ganoushoreilly »

elorant wrote: post #140

The motherboard has 7 PCie slots and there are 8 GPUs. So where does the spare one connect to? Is he using two GPUs in the same slot limiting the bandwidth?

may be using an nvme to pcie adapter, common in the crypto mining world

Re: Serving AI from the Basement – 192GB of VRAM Setup

#144

Post by Dylan16807 »

NavinF wrote: post #141
Dylan16807 wrote: post #86

Earlier quoted context omitted.

A standard outlet is 15A, and you're only allowed to use 80% continuously. So now you're dropping to 150W each. Using two circuits is viable.

In the bay area every home I've lived in had 20A circuits for everything. The NEC 80% rule is only relevant in data centers where someone enforces it. The time-current curve for any 20A breaker shows that it will take infinite time to trip when you draw 20A. This is correct behavior because 12awg wire can handle 20A indefinitely

Do they actually install the 20A outlets with the sideways part? I don't know if I've ever seen one in person.

I guess the outlet doesn't matter a whole lot, especially if you take multiple cables to the wall.

But I still don't want to be doing unenforced electrical violations.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#145

Post by smcnally »

J_Shelby_J wrote: post #100
mattnewton wrote: post #50

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

I’m running two 3090s on a 700w psu. You definitely can get more than that out of 2000w bus. I wrote a blog on reducing the power limits of nvidia gpus. Definitely try it out. https://shelbyjenkins.github.io/blog/power-limit-nvidia-linu...

Thank you for this post. I’d read it in ~June and it helped quite a bit with manual ‘nvidia-smi’ runs. I just recently created the systemd service description and am still delving related power and performance possibilities.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#146

Post by Dylan16807 »

wkat4242 wrote: post #80
thelastparadise wrote: post #75

Earlier quoted context omitted.

> It's already really hard to buy big HDDs for my NAS IME 20tb drives are easy to find. I don't think the clouds have access to bigger drives or anything. Similarly, we can buy 8x A100s, they're just fundamentally expensive whether you're a business or not. There doesn't seem to be any "wall" up like there used to be with proprietary hardware.

They are easy to find but extremely expensive. I used to pay below 200€ for a 14TB Seagate 8 years ago. That's now above 300. And the bigger ones are even more expensive. For me these prices are prohibitive. Just like the A100s are (though those are even more so of course). The problem is the common consumer relying on the cloud so these kind of products become niches and lose volume. Also, the cloud providers don't…

Well if I look at Amazon I see a couple models of external 14TB for $190, and a brand new Exos 16TB for $230. Not too bad. Though personally I get much cheaper used drives and put them in RAID for a NAS.

And they do have better sales.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#147

Post by fuzzybear3965 »

disiplus wrote: post #123
nullindividual wrote: post #46

Earlier quoted context omitted.

The reason I asked is I used to run a dual X5650 server with SSDs and it was about $50/month with the cheapest (or very close to) rates in the US.

We have more expensive gas then usa, but i pay like a 5 cent per kwh @220V Did not know how expensive it is in usa, especially California.

Yep. ~$.33/kWh in Southern California (SoCal Edison) and going up all the time!

Re: Serving AI from the Basement – 192GB of VRAM Setup

#148

Post by lbotos »

varispeed wrote: post #78
3eb7988a1663 wrote: post #73

Earlier quoted context omitted.

People can and do die from misuses of electricity. Not a move-fast-and-break things kind of domain.

You only "break" once...

I've been learning Japanese and a favorite of mine is: 一体

Which is used as "what the heck" but it's direct kanji translation is one body.

https://jisho.org/word/%E4%B8%80%E4%BD%93

Re: Serving AI from the Basement – 192GB of VRAM Setup

#149

Post by lbotos »

flixf wrote: post #67

Very interesting! How are the 8 GPUs connected to the motherboard? Based on the article and the pictures, he doesn't appear to be using PCIe risers. I have a setup with 3 RTX 3090 GPUs and the PCIe risers are a huge source of pain and system crashes.

I had the same question. I was curious what retimers he was using.

I've had my eye on these for a bit https://c-payne.com/

Re: Serving AI from the Basement – 192GB of VRAM Setup

#150

Post by fragmede »

fuzzybear3965 wrote: post #147
disiplus wrote: post #123

Earlier quoted context omitted.

We have more expensive gas then usa, but i pay like a 5 cent per kwh @220V Did not know how expensive it is in usa, especially California.

Yep. ~$.33/kWh in Southern California (SoCal Edison) and going up all the time!

$0.51/KwH during peek hours in San Francisco!
Post reply on HN