Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

41–50 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#41

Post by philsnow »

jrflowers wrote: post #29
NavinF wrote: post #23

Earlier quoted context omitted.

You could power limit the 3090s to fit a standard 120V*20A = 2400W outlet if you really want to. The default power limit is 350W each so you'll only lose a little perf. Also most rooms have multiple circuits. Just connect half the GPUs to each outlet. I already do this with my desktop PC because it has 2 PSUs. Also most homes in the US have 30A*240V = 7200W dryer/stove outlets in the kitchen, laundry room, garage, et…

Brb replacing my stove with a bunch of GPUs

Easy-bake cryptominer

Re: Serving AI from the Basement – 192GB of VRAM Setup

#42

Post by bravura »

XMasterrrr wrote: post #2

Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…

How loud is it? Was special electrical needed?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#43

Post by cranberryturkey »

angoragoats wrote: post #10
rvnx wrote: post #7

You could just buy a Mac Studio for 6500 USD, have 192 GB of unified RAM and have way less power consumption.

You could for sure, but the nVidia setup described in this article would be many times faster at inference. So it’s a tradeoff between power consumption and performance. Also, modern GPUs are surprisingly good at throttling their power usage when not actively in use, just like CPUs. So while you need 3kW+ worth of PSU for an 8x3090 setup, it’s not going to be using anywhere near 3kW of power on average, unless you’re…

Can Reflection:70b work on them?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#44

Post by modeless »

I wonder how the cost compares to a Tinybox. $25k for 6x 4090 or $15k for 6x 7900XTX. Of course that's the full package with power supplies, CPU, storage, cooling, assembly, shipping, etc. And a tested, known good hardware/software configuration which is crucial with this kind of thing.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#45

Post by angoragoats »

wkat4242 wrote: post #32
angoragoats wrote: post #11

Earlier quoted context omitted.

Yep. The 8x3090 setup should be at least 10 times faster than the Mac.

Yes though for inference memory bandwidth is more important than tflops. Not sure how Apple compares in that regard. The OP does train models too though which is more compute heavy.

Each of the 3090s has something like 15% higher memory bandwidth than the whole M2 Ultra system does.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#46

Post by nullindividual »

michaelt wrote: post #39
nullindividual wrote: post #5

Earlier quoted context omitted.

Do you run this 24/7? What is your cost of electricity per kilowatt hour and what is the cost of this setup per month?

I have a much smaller setup than the author - a quarter the GPUs and RAM - and I was surprised to find it draws 300W at idle

The reason I asked is I used to run a dual X5650 server with SSDs and it was about $50/month with the cheapest (or very close to) rates in the US.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#47

Post by angoragoats »

evilduck wrote: post #14
sva_ wrote: post #9

Earlier quoted context omitted.

A single 3090 will deliver more tflops than the m2 ultra.

The M2 Ultra doesn't require doing electrical work on your house like this 8x 3090 setup did though.

You can run 8 or even more 3090s without doing electrical work on a standard US residential power feed, you just need outlets connected to multiple circuits. This is not always convenient, which is why some may choose to do electrical work, but it’s absolutely not required.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#48

Post by michaelt »

NavinF wrote: post #27
3eb7988a1663 wrote: post #12

What is the power draw under load/idle? Does it noticeably increase the room temperature? Given the surroundings (aka the huge pile of boxes behind the setup), curious if you could get away with just a couple of box fans instead of the array of case fans. Are you intending to use the capacity all for yourself or rent it out to others?

Box fans are surprisingly power hungry. You'd be better off using large 200mm PC fans. They're also a lot quieter

If you care about noise, I also recommend not getting 8 GPUs with 3 fans each :)

Re: Serving AI from the Basement – 192GB of VRAM Setup

#49

Post by angoragoats »

cranberryturkey wrote: post #43
angoragoats wrote: post #10

Earlier quoted context omitted.

You could for sure, but the nVidia setup described in this article would be many times faster at inference. So it’s a tradeoff between power consumption and performance. Also, modern GPUs are surprisingly good at throttling their power usage when not actively in use, just like CPUs. So while you need 3kW+ worth of PSU for an 8x3090 setup, it’s not going to be using anywhere near 3kW of power on average, unless you’re…

Can Reflection:70b work on them?

Maybe you meant to reply to a different comment? Work on what?

Edit: I guess to directly answer your question, I don’t see why you couldn’t run a 70b model at full quality on either a M2 192GB machine or on an 8x 3090 setup.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#50

Post by mattnewton »

XMasterrrr wrote: post #2

Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
Post reply on HN