this is why we need an actual AI blockchain, so we can donate GPU and earn rewards for the p2p api calls using the distributed model.
Problem is once you have to scale to multiple GPUs the interconnect becomes the primary bottleneck.
Serving AI from the Basement – 192GB of VRAM Setup
91–100 of 279 posts
Re: Serving AI from the Basement – 192GB of VRAM Setup
#92this is why we need an actual AI blockchain, so we can donate GPU and earn rewards for the p2p api calls using the distributed model.
That's actually interesting. While crypto GPU mining is "purposeless" or arbitrary, would be way cooler if to GPU mine meant to chunk through computing tasks in a free/open queue (blockchain). Eventually there could be some tipping point where networks are fast enough and there are enough hosting participants it could be like a worldwide/free computing platform - not just for AI for anything.
I also think this idea has been explored a little bit at least in terms of GPU distribution networks for AI (Petal and Horde come to mind).
Re: Serving AI from the Basement – 192GB of VRAM Setup
#93You could just buy a Mac Studio for 6500 USD, have 192 GB of unified RAM and have way less power consumption.
I'm seeing this misunderstanding a lot recently. There's TWO components to putting together a viable machine learning rig:
- Fitting models in memory
- Inference / Training speed
8 x RTX 3090s will absolutely CRUSH a single Mac Studio in raw performance.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#94Earlier quoted context omitted.
The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.
A good engineer knows the difference between safe and dangerous. Setting up an AI computer is safe. Maybe you trip a circut. Maybe you interfere with something else running on your hobby computer. But nothing bad can really happen.
Residential electrical is dangerous. Maybe you electrocute yourself. Maybe you cause a fire 5 years down the line. Maybe you cause a fire for the next owner because you didn't know to protect the wire with a metal plate so they drill into it.
Having said that, 2 4090s will run you aroud $5,000, not counting any of the surrounding system. At that cost point, hireing an electritian would not be that big of an expense relativly speaking.
Also, if you are at the point where you need to add a circut for power, you might need to seriously consider cooling, which could potentially be another side quest.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#95Earlier quoted context omitted.
[flagged]
He's got 8x3090s are you fucking kidding? Like is this some kind of AI reply? "Wow great post! I enjoy your valuable contributions. Can you tell me more about graphics cards and how they compare to other different types of computers? I am interested and eager to learn! :)"
kind of violent approach but I agree on the bottom line. I don't see why should somebody be enthousiast about this. someone was just able to spend 8× the figures a random teen is able to spend for his gaming rig, and he just iterated the teen's rig 8 times, he then installed ubuntu+cuda and called it a day.
something that is actually interesting that is attempting to bring something on the table : check tinygrad/tinycorp
Re: Serving AI from the Basement – 192GB of VRAM Setup
#96Re: Serving AI from the Basement – 192GB of VRAM Setup
#97Earlier quoted context omitted.
The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
Not speaking from direct experience building a rig like this but the blog post mentions having 3 power supplies so the most direct solution would be to put each on their own dedicated circuit. As long as you have space in your electrical box this is straightforward to do though I would recommend having an electrician do the wiring if you aren’t experienced with that type of home electrical work.
Even without space in the existing box, installing a subpanel isn't that much more of a cost.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#98I have a similar setup in my basement! Although its multiple nodes, with a total of 16x3090s. Also needed to install a 30A 240V circuit as well.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#99Very interesting! How are the 8 GPUs connected to the motherboard? Based on the article and the pictures, he doesn't appear to be using PCIe risers. I have a setup with 3 RTX 3090 GPUs and the PCIe risers are a huge source of pain and system crashes.
Looks like SlimSAS.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#100Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…
The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
I’m running two 3090s on a 700w psu. You definitely can get more than that out of 2000w bus.
I wrote a blog on reducing the power limits of nvidia gpus. Definitely try it out. https://shelbyjenkins.github.io/blog/power-limit-nvidia-linu...