Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

201–210 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#201
post #124
post #105

Earlier quoted context omitted.

Add to that is that it is likely illegal to do yourself. Which of course has implications for insurance etc.

In the US, it’s fully legal to perform electric/plumbing/whatever work on your own home. If you screw it up and need to file a claim, insurance can’t deny the claim based solely on the fact that you performed the work yourself, even if you’re not a certified electrician/plumber/whatever. What you don't want to do is have an unlicensed friend work on your home, and vice versa. There are no legal protections, and the i…

Around here, the bar is lower for work on your own property, but you still need to be qualified by the county to be allowed to do so. Qualification consists of a 2 hour open book exam, where the book is a copy of the national electrical codes.

https://www.montgomerycountymd.gov/DPS/Process/combuild/home...

Granted, if you actually do unlicensed work in your house, no one will know. But it is still illegal.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#202

Amazing writeup. And what a heavy hitter of an inaugural blog entry... This might be the right time to ask: So, on the one hand, this is what it takes to pack 192gb of Nvidia flavored vram into a home server. I'm curious, is there any hope of doing any interesting work on a MacBook Pro Which currently can be max-spaced at 128 GB of unified memory (for the low, low price of $4.7k). I know there's no hope of running cu…

Download kobodlcpp and give it a try. It is a single exec and uses metal acceleration with an Apple Arm CPU.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#203

Earlier quoted context omitted.

The reason I asked is I used to run a dual X5650 server with SSDs and it was about $50/month with the cheapest (or very close to) rates in the US.

We have more expensive gas then usa, but i pay like a 5 cent per kwh @220V Did not know how expensive it is in usa, especially California.

Depends on where you live in the U.S. On the west coast, you’ll definitely pay 2x the national average. I live in the south-eastern part of the U.S and electricity is quite a bit cheaper at 12 cents/kwh. But even here it varies by region in my state of Georgia. Down the road where I have friends and family , they get power from a different provider. Their cost are much higher. But they also have flexible pricing. Such as at night they pay a fraction of their day time rates. From 17 cents to 3 cents. While I have a fixed rate 24/7. Which is cheaper than their day time rates.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#205
post #168

Typical crypto miner setup. I had two 6GPU setups with 1200W PSUs and 6 PCIE slots with PCI extender cables. Its value dropped harder than a cyber truck's after a few months. The worst thing is dust. They would accumulate so much every week I had to blow the dust off with an air compressor. Electricity cost was around $4 a day (24 x $0.20~). If online GPU renting is more expensive, maybe the initial cost could be jus…

> Typical crypto miner setup. Except not doing the sketchy x1 pcie lanes. That’s the part that makes nice LLM setups hard

Can you tell me what's sketchy about it? I have not had an issue with any one of the 12 extenders and bandwidth held well without any issues. Please explain if possible if LLM requires a different type of extender.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#206

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

You can run a setup of 8x 4090 GPUs using 4x 1200W 240V power supplies (preferably HP HSTNS-PD30 Platinum Series), with a collective use of just around 20-amps, meaning it can easily run on a single 240V 20-amp breaker. This should be easily doable in a home where you typically have a 100 to 200A main power panel. Running 4x 1200W power supplies 24 hours a day will consume 115.2 kWh per day. At an electricity rate of…

In Cali isn't now like 0.5 per kWh :P

Re: Serving AI from the Basement – 192GB of VRAM Setup

#207
post #4

How much do the NVLinks help in this case? Do you have a rough estimate of how much this cost? I'm curious since I just built my own 2x 3090 rig and I wondered about going EPYC for the potential to have more cards (stuck with AM5 for cheapness though). All in all I spent about $3500 for everything. I'm guessing this is closer to $12-15k? CPU is around $800 on eBay.

I haven't been able to find a good answer on what difference NVLink makes or which applications support it.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#208
post #155

Earlier quoted context omitted.

Ah yes, the "move fast and burn your house down" school of "engineering".

Adding a bog-standard breaker and a short conduit run is about as simple as it gets for electric work. It’s rather low risk if you simply read the code and follow it. If you know nothing about basic electric work or principles, sure - spend the $500 to have an electrician add a 30 or 50A 220V outlet near your electric service panel. Totally reasonable to do as it is indeed dangerous to touch things you don’t understa…

I'm not saying people shouldn't add breakers. I'm saying your talking like people are scaredy-cats and comparing it to working with toys or hello world is exactly the kind of of macho nonsense that leads people to do shoddy engineering.

As a hobby, I restore pinball machines. A modern one is extremely careful about how it uses power, limiting wall current to a small, normally-sealed section of the machine. And even so, it automatically disables the lower-voltage internals the moment you open the coin door. A 1960s machine, by contrast, may not have a ground at all. It may have an unpolarized plug, and it will run wall current all over the place, including the coin door, one of the flippers, and a mess of relays.

In the pinball community, you'll find two basic attitudes toward this. One is people treating electrical safety about as seriously as the people who design the modern machines. The others is people who think anybody who worries about a little wall current are all pussies who don't have the balls to work on anything and should just man up and not worry about a little 120V jolt.

The truth is that most people here are not engineers of any sort. We're software developers. We're used to working in situations where safety and rigor basically don't matter, where you have to just cowboy ahead and try shit. And that's fine, because control-z is right there. I've met people who bring that attitude to household electrical work, and they're fucking dangerous. I know one guy, quite a smart one, who did a lot of his own electrical work based on manliness and arrogance, and once the inspector caught up with him, he immediately pulled the guy's meter and wouldn't let him connect up to the grid again until a real electrician had straightened it all out.

It's true that this stuff is not that hard to learn if you study it. But an architect friend likes to say that the building code is written in blood, meaning that much of it is there because confident dumbasses managed to kill enough people that they had to add a new rule. If people are prepared to learn the rules and appreciate why they're there, I'm all for it. But if they do it coming from a place of proving that they're not "so afraid of residential power", that's a terrible way to approach it.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#209
post #4

How much do the NVLinks help in this case? Do you have a rough estimate of how much this cost? I'm curious since I just built my own 2x 3090 rig and I wondered about going EPYC for the potential to have more cards (stuck with AM5 for cheapness though). All in all I spent about $3500 for everything. I'm guessing this is closer to $12-15k? CPU is around $800 on eBay.

I haven't been able to find a good answer on what difference NVLink makes or which applications support it.

NVLink is what makes multiGPU work. It lets the GPUs talk to each other across a high bandwidth (600 Gbps), low latency link. Tensorflow and PyTorch both support it, among other things. It's not this weird thing that's a side note, the interconnect between nodes is what makes a supercomputer super. You don't hear about it much because you don't hear about a lot of details of supercomputer stuff in mainstream media.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#210

Amazing writeup. And what a heavy hitter of an inaugural blog entry... This might be the right time to ask: So, on the one hand, this is what it takes to pack 192gb of Nvidia flavored vram into a home server. I'm curious, is there any hope of doing any interesting work on a MacBook Pro Which currently can be max-spaced at 128 GB of unified memory (for the low, low price of $4.7k). I know there's no hope of running cu…

Download kobodlcpp and give it a try. It is a single exec and uses metal acceleration with an Apple Arm CPU.

Thanks!
Post reply on HN