You could just buy a Mac Studio for 6500 USD, have 192 GB of unified RAM and have way less power consumption.
[flagged]
Serving AI from the Basement – 192GB of VRAM Setup
131–140 of 279 posts
Re: Serving AI from the Basement – 192GB of VRAM Setup
#132You could just buy a Mac Studio for 6500 USD, have 192 GB of unified RAM and have way less power consumption.
[flagged]
Re: Serving AI from the Basement – 192GB of VRAM Setup
#133Earlier quoted context omitted.
> I can't believe a group of engineers are so afraid of residential power. ... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise. The instinct to not touch something that you don't yet deeply understand is very much an engineer's instinct. Any engineer worthy of the title has often spent weeks carefully designin…
Sir, this is "Hacker News".
So did you mean "I can't believe a group of hackers are so afraid of residential power"?
Re: Serving AI from the Basement – 192GB of VRAM Setup
#134> And who knows, maybe someone will look back on my work and be like “haha, remember when we thought 192GB of VRAM was a lot?” I wonder if this will happen. It's already really hard to buy big HDDs for my NAS because nobody buys external drives anymore. So the pricing has gone up a lot for the prosumer. I expect something similar to happen to AI. The big cloud parties are all big leaders on LLMs and their goal is to…
>It's already really hard to buy big HDDs for my NAS because nobody buys external drives anymore. So the pricing has gone up a lot for the prosumer.
I buy refurb/used enterprise drives for that reason, generally around $12 per TB for the recent larger drives. And around $6 per TB for smaller drives. You just need an SAS interface but that's not difficult or expensive.
IE; 25TB for $320, or 12TB for $80.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#135Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…
The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
I’ve ran 3x L40S on a 1650W PSU on a normal 120V 20A circuit.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#136So, how do you connect the 8th card if you have 7 PCIe 4.0 x16 slots available?
PCIe bifurcation - so splitting one of the x16 slots into two x8 or similar.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#137Earlier quoted context omitted.
Americans do not have electric kettles and need special circuits for electric clothes dryers.
We have an electric kettle in the US and it runs just fine drawing 1500W. You're correct that the dryer is on a larger circuit, though.
> and it runs just fine drawing 1500W.
You think that this is "just fine" because you've never experienced the glory that is a 3kW kettle!
Re: Serving AI from the Basement – 192GB of VRAM Setup
#138Earlier quoted context omitted.
Americans do not have electric kettles and need special circuits for electric clothes dryers.
We have an electric kettle in the US and it runs just fine drawing 1500W. You're correct that the dryer is on a larger circuit, though.
I get bored and tend to wander off waiting for it to boil at 3kW
1.5kW must be absolute agony
Re: Serving AI from the Basement – 192GB of VRAM Setup
#139Earlier quoted context omitted.
What? No I just don’t know the difference, sorry. I am interested in learning more about running 405b parameter models, which I believe you can do on a 192gb M series Mac. The answer here is that the Nvidia system has much better performance. I’ve been focused on “can I even run the model” I didn’t think about the actual performance of the system.
It's kinda hard to believe that someone would stumble onto the landmine of AI performance comparison between Apple Silicon and Nvidia hardware. People are going to be rude because this kinda behavior is genuinely indistinguishable from bad-faith trolling. From benchmarks alone, you can easily tell that the performance-per-watt of any Mac Studio gets annihilated by a 4090: https://browser.geekbench.com/opencl-benchmar…
> From benchmarks alone, you can easily tell that the performance-per-watt of any Mac Studio gets annihilated by a 4090: https://browser.geekbench.com/opencl-benchmarks
The Geekbench GPU compute benchmarks are nearly worthless in any context, and most certainly are useless for evaluating suitability for running LLMs, or anything involving multiple GPUs.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#140The motherboard has 7 PCie slots and there are 8 GPUs. So where does the spare one connect to? Is he using two GPUs in the same slot limiting the bandwidth?