Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

161–170 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#161

Post by koyote »

XMasterrrr wrote: post #2

Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…

This is undoubtedly cool and I am a bit jealous!

Maybe a bit of a stupid question, but what do you actually do with the models you run/build, a part from tinkering? I'd assume most tinkering can also be done on smaller systems? Is it in order to build a model that is actually 'useful'/competitive?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#162

Post by system2 »

Typical crypto miner setup. I had two 6GPU setups with 1200W PSUs and 6 PCIE slots with PCI extender cables. Its value dropped harder than a cyber truck's after a few months.

The worst thing is dust. They would accumulate so much every week I had to blow the dust off with an air compressor.

Electricity cost was around $4 a day (24 x $0.20~). If online GPU renting is more expensive, maybe the initial cost could be justifiable.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#163

Post by walterbell »

wkat4242 wrote: post #80
thelastparadise wrote: post #75

Earlier quoted context omitted.

> It's already really hard to buy big HDDs for my NAS IME 20tb drives are easy to find. I don't think the clouds have access to bigger drives or anything. Similarly, we can buy 8x A100s, they're just fundamentally expensive whether you're a business or not. There doesn't seem to be any "wall" up like there used to be with proprietary hardware.

They are easy to find but extremely expensive. I used to pay below 200€ for a 14TB Seagate 8 years ago. That's now above 300. And the bigger ones are even more expensive. For me these prices are prohibitive. Just like the A100s are (though those are even more so of course). The problem is the common consumer relying on the cloud so these kind of products become niches and lose volume. Also, the cloud providers don't…

Looking at https://shucks.top and https://diskprices.com, prices do seem to be higher.

Homelab vendor in Austin, TX with periodic sales, limited volume: https://shop.digitalspaceport.com

Re: Serving AI from the Basement – 192GB of VRAM Setup

#164

Post by buildbot »

lvl155 wrote: post #110
choilive wrote: post #98

I have a similar setup in my basement! Although its multiple nodes, with a total of 16x3090s. Also needed to install a 30A 240V circuit as well.

That last part is often overlooked. This is also why sometimes it’s just not worth going local especially if you don’t need all that compute power beyond a few days.

100% agree, anything beyond 4x gpu’s is getting into the very annoying to power territory and makes the cloud very attractive. I already can trip a 15A circuit on 115v power with just 3x4090s and a SPR-X cpu.

It also costs a lot to power. In the summer, 2x more than you expect, because unless it’s outside, you need cool 1000+ watts of extra heat with your AC. All that together and runpod starts to look very tempting!

Re: Serving AI from the Basement – 192GB of VRAM Setup

#165

Post by oceanplexian »

Eisenstein wrote: post #89
lvl155 wrote: post #52

Earlier quoted context omitted.

My reason for going Epyc was for Pcie lanes and cheaper enterprise SSDs via U.3/2. With AM5, you tap out the lanes with dual GPUs. Threadripper is preferable but Epyc is about 1/2 of the price or even better if you go last gen.

Why do you need such high cross card bandwidth for inference? Are you hosting for a lot of users at once?

The Epyc boards make things way easier (I have 4 epyc boards of various generations) because they have loads of x16 slots and you’re not screwing around with bifurcation and sketchy PCI splitters. Another oft-forgotten item that consumes lanes is 25 or 40Gb NICs which you might fine you want if you’re pushing big model files around to other machines or storage.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#166

Post by vunderba »

gizmo686 wrote: post #94
tcdent wrote: post #70

Earlier quoted context omitted.

I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

A good engineer knows the difference between safe and dangerous. Setting up an AI computer is safe. Maybe you trip a circut. Maybe you interfere with something else running on your hobby computer. But nothing bad can really happen. Residential electrical is dangerous. Maybe you electrocute yourself. Maybe you cause a fire 5 years down the line. Maybe you cause a fire for the next owner because you didn't know to prot…

Just a slight clarification, an RTX 4090 card currently runs about $1700 USD at least in the states, so it's more like $3500 pre-tax.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#167

Post by scubbo »

m-s-y wrote: post #124
tjoff wrote: post #105

Earlier quoted context omitted.

Add to that is that it is likely illegal to do yourself. Which of course has implications for insurance etc.

In the US, it’s fully legal to perform electric/plumbing/whatever work on your own home. If you screw it up and need to file a claim, insurance can’t deny the claim based solely on the fact that you performed the work yourself, even if you’re not a certified electrician/plumber/whatever. What you don't want to do is have an unlicensed friend work on your home, and vice versa. There are no legal protections, and the i…

Extraordinary claims require extraordinary evidence.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#168

Post by Havoc »

system2 wrote: post #162

Typical crypto miner setup. I had two 6GPU setups with 1200W PSUs and 6 PCIE slots with PCI extender cables. Its value dropped harder than a cyber truck's after a few months. The worst thing is dust. They would accumulate so much every week I had to blow the dust off with an air compressor. Electricity cost was around $4 a day (24 x $0.20~). If online GPU renting is more expensive, maybe the initial cost could be jus…

> Typical crypto miner setup.

Except not doing the sketchy x1 pcie lanes. That’s the part that makes nice LLM setups hard

Re: Serving AI from the Basement – 192GB of VRAM Setup

#170

Post by bigiain »

gizmo686 wrote: post #94
tcdent wrote: post #70

Earlier quoted context omitted.

I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

A good engineer knows the difference between safe and dangerous. Setting up an AI computer is safe. Maybe you trip a circut. Maybe you interfere with something else running on your hobby computer. But nothing bad can really happen. Residential electrical is dangerous. Maybe you electrocute yourself. Maybe you cause a fire 5 years down the line. Maybe you cause a fire for the next owner because you didn't know to prot…

> Also, if you are at the point where you need to add a circut for power, you might need to seriously consider cooling, which could potentially be another side quest.

There should be an easy/reliable way to channel "waste heat" from something like this to your hot water system.

Actually, 4 or 5 kW continuous is a lot more than most domestic hot water services need. So in my usual manner of overcomplicating simple ideas, now I want to use the waste heat to run a boiler driving a steam engine, perhaps to directly mechanically run your air conditioning or heat pump compressor.

Post reply on HN