Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

121–130 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#121

Post by wpietri »

tcdent wrote: post #70
mattnewton wrote: post #50

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

Ah yes, the "move fast and burn your house down" school of "engineering".

Re: Serving AI from the Basement – 192GB of VRAM Setup

#122

Post by angoragoats »

modeless wrote: post #44

I wonder how the cost compares to a Tinybox. $25k for 6x 4090 or $15k for 6x 7900XTX. Of course that's the full package with power supplies, CPU, storage, cooling, assembly, shipping, etc. And a tested, known good hardware/software configuration which is crucial with this kind of thing.

You can build a setup like in the OP for somewhere around $10k, depending on several factors, the most important of which are the price you source your GPUs at ($700 per 3090 is a reasonable going rate) and what CPU you choose (high core count, high frequency Epyc CPUs will cost more).

Re: Serving AI from the Basement – 192GB of VRAM Setup

#123

Post by disiplus »

nullindividual wrote: post #46
michaelt wrote: post #39

Earlier quoted context omitted.

I have a much smaller setup than the author - a quarter the GPUs and RAM - and I was surprised to find it draws 300W at idle

The reason I asked is I used to run a dual X5650 server with SSDs and it was about $50/month with the cheapest (or very close to) rates in the US.

We have more expensive gas then usa, but i pay like a 5 cent per kwh @220V

Did not know how expensive it is in usa, especially California.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#124

Post by m-s-y »

tjoff wrote: post #105
gizmo686 wrote: post #94

Earlier quoted context omitted.

A good engineer knows the difference between safe and dangerous. Setting up an AI computer is safe. Maybe you trip a circut. Maybe you interfere with something else running on your hobby computer. But nothing bad can really happen. Residential electrical is dangerous. Maybe you electrocute yourself. Maybe you cause a fire 5 years down the line. Maybe you cause a fire for the next owner because you didn't know to prot…

Add to that is that it is likely illegal to do yourself. Which of course has implications for insurance etc.

In the US, it’s fully legal to perform electric/plumbing/whatever work on your own home.

If you screw it up and need to file a claim, insurance can’t deny the claim based solely on the fact that you performed the work yourself, even if you’re not a certified electrician/plumber/whatever.

What you don't want to do is have an unlicensed friend work on your home, and vice versa. There are no legal protections, and the insurance companies absolutely will go after you/your friend for damages.

Edit: sorry this applies to owned property, not if you’re renting

Re: Serving AI from the Basement – 192GB of VRAM Setup

#127

Post by lossolo »

XMasterrrr wrote: post #2

Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…

Cool, it looks similar to my crypto mining rigs (8xGPU per node) from around 7 years ago, but I used PCI-E risers and a dual power supply.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#128

Post by 51290046 »

exyi wrote: post #54
angoragoats wrote: post #10

Earlier quoted context omitted.

You could for sure, but the nVidia setup described in this article would be many times faster at inference. So it’s a tradeoff between power consumption and performance. Also, modern GPUs are surprisingly good at throttling their power usage when not actively in use, just like CPUs. So while you need 3kW+ worth of PSU for an 8x3090 setup, it’s not going to be using anywhere near 3kW of power on average, unless you’re…

Even if you are running it constantly, the per token power consumption is likely going to be in a similar range, not to mention you'd need 10+ macs for the throughput.

[flagged]

Re: Serving AI from the Basement – 192GB of VRAM Setup

#129

Post by tcdent »

lolinder wrote: post #103
tcdent wrote: post #70

Earlier quoted context omitted.

I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

> I can't believe a group of engineers are so afraid of residential power. ... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise. The instinct to not touch something that you don't yet deeply understand is very much an engineer's instinct. Any engineer worthy of the title has often spent weeks carefully designin…

Sir, this is "Hacker News".
Post reply on HN