Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

261–270 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#261

Post by fennecbutt »

pupdogg wrote: post #192
swader999 wrote: post #191

Earlier quoted context omitted.

You'll have helicopters over your house.

Believe it or not, a buddy of mine was dealing with crazy high power bills for three months and couldn’t figure out why. He tried everything to cut back, but nothing worked. Finally, he called an electrician, who found a hidden short in a 30-amp circuit that was constantly drawing power without tripping the breaker. After fixing the issue, his bills went back to normal—no helicopters were involved during the process!

And all that heat was going...where?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#262

Post by throwthrowuknow »

rspoerri wrote: post #234
throwthrowuknow wrote: post #71

Earlier quoted context omitted.

Not going to work for training from scratch which is what the author is doing.

192GByte of RAM are not enough to train 405B models. Reflection 70B requires 140GByte of RAM in fp16, 405 would need ~810Gbyte of RAM.

Pretty sure he said he’s inferencing llama3 405 and training his own custom model from scratch. He didn’t say how big his custom model will be.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#263

Post by throwthrowuknow »

XMasterrrr wrote: post #222
throwthrowuknow wrote: post #65

Earlier quoted context omitted.

Not speaking from direct experience building a rig like this but the blog post mentions having 3 power supplies so the most direct solution would be to put each on their own dedicated circuit. As long as you have space in your electrical box this is straightforward to do though I would recommend having an electrician do the wiring if you aren’t experienced with that type of home electrical work.

Actually, putting each PSU on its own circuit is crazy dangerous. In the scenario of your suggestion, if one goes out, you are in for a fire. Highly recommend against that.

Do they not have fuses? Use 15A breakers?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#264

Post by angoragoats »

metadat wrote: post #252
angoragoats wrote: post #247

Earlier quoted context omitted.

While you're technically correct, assuming you're using PCIe 4.0 or higher, the performance difference between x8 and x16 is practically zero.

Is this because the graphics cards are not using all available PCIe bandwidth? Or why?

Yes. In fact, even running a video card in a 4x slot (again, assuming PCIe 4.0) results in only a modest (5-20%, depending on what you’re doing) drop in speeds.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#265

Post by Yodel0914 »

XMasterrrr wrote: post #223
Yodel0914 wrote: post #184

Earlier quoted context omitted.

I'm pretty sure in our current rental the kitchen and laundry are on the same circuit, which means I'll often have the dryer, washing machine, kettle, toaster and microwave drawing power at the same time. It's never been an issue.

What kind of a dryer? Because that cannot be right. Dryers require a 30amp 240volt dedicated breakers by code in most counties in most states nowadays.

It's a bosch heat pump dryer, but previously we had a traditional vented dryer.

I've never seen a dedicated circuit for dryers in Australia, and I've lived in probably a dozen different properties. Ovens, aircon, hot water, bathroom heat lamps often have dedicated circuits, though.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#266

Post by halJordan »

Tepix wrote: post #112
modeless wrote: post #44

I wonder how the cost compares to a Tinybox. $25k for 6x 4090 or $15k for 6x 7900XTX. Of course that's the full package with power supplies, CPU, storage, cooling, assembly, shipping, etc. And a tested, known good hardware/software configuration which is crucial with this kind of thing.

If you merely want CUDA and lots of VRAM there‘s no reason to pick expensive 4090s over used 3090s

Well there is and it's called performance. You dont have to push your version of what an appropriate price/performance ratio is

Re: Serving AI from the Basement – 192GB of VRAM Setup

#267

Post by dymk »

rootusrootus wrote: post #242
dymk wrote: post #199

Earlier quoted context omitted.

Because they don’t have a spare few grand for an instant hot plus installation

You’re off by an order of magnitude. They are a couple hundred bucks and an easy DIY job.

It's a couple hundred bucks if you can DIY it, which most people cannot or are not willing to do

Re: Serving AI from the Basement – 192GB of VRAM Setup

#268

Post by choilive »

buildbot wrote: post #221
choilive wrote: post #211

Earlier quoted context omitted.

Getting that circuit installed was pretty cheap likely because its in an unfinished and unconditioned basement. The basement stays comfortable even during the summer. The heat does seem to work its way into the rest of the house but the additional cooling load is only about 20% more than usual. It lowers the heating cost about the same amount during the winter so it works itself out.

Yeah location of your place, climate, and placement of the server in the house will affect this a lot. I'm on the top story of a building, even in the winter I rarely need to turn on my heat, just getting by on the waste heat of the rest of the building. My assortment of machines will easily keep the living room at 25C+ with a window open unless it's below 10C out! If I could keep the servers in the cool parking gara…

Yeah I worked it out and I am saving ~75% vs running my inference workloads on RunPod. $650/mo in electricity vs $2,500/mo to do the same thing on RunPod. Been in near continuous operation over 9 months, so the system has basically paid for itself with the savings.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#270

Post by matt-p »

wpietri wrote: post #208
phil21 wrote: post #155

Earlier quoted context omitted.

Adding a bog-standard breaker and a short conduit run is about as simple as it gets for electric work. It’s rather low risk if you simply read the code and follow it. If you know nothing about basic electric work or principles, sure - spend the $500 to have an electrician add a 30 or 50A 220V outlet near your electric service panel. Totally reasonable to do as it is indeed dangerous to touch things you don’t understa…

I'm not saying people shouldn't add breakers. I'm saying your talking like people are scaredy-cats and comparing it to working with toys or hello world is exactly the kind of of macho nonsense that leads people to do shoddy engineering. As a hobby, I restore pinball machines. A modern one is extremely careful about how it uses power, limiting wall current to a small, normally-sealed section of the machine. And even s…

To be fair I'd be quite a bit more relaxed working on 120v. Very supprised these machines don't run on dc internally?
Post reply on HN