[flagged]
Serving AI from the Basement – 192GB of VRAM Setup
181–190 of 279 posts
Re: Serving AI from the Basement – 192GB of VRAM Setup
#182Earlier quoted context omitted.
He's got 8x3090s are you fucking kidding? Like is this some kind of AI reply? "Wow great post! I enjoy your valuable contributions. Can you tell me more about graphics cards and how they compare to other different types of computers? I am interested and eager to learn! :)"
What? No I just don’t know the difference, sorry. I am interested in learning more about running 405b parameter models, which I believe you can do on a 192gb M series Mac. The answer here is that the Nvidia system has much better performance. I’ve been focused on “can I even run the model” I didn’t think about the actual performance of the system.
You might enjoy last fortnight's The Register article:
Buying a PC for local AI? These are the specs that matter
https://www.theregister.com/2024/08/25/ai_pc_buying_guide/
It died without interest here: https://news.ycombinator.com/item?id=41347785 likely time of day, I'm mostly active in HN off peak hours.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#183Earlier quoted context omitted.
In the US, it’s fully legal to perform electric/plumbing/whatever work on your own home. If you screw it up and need to file a claim, insurance can’t deny the claim based solely on the fact that you performed the work yourself, even if you’re not a certified electrician/plumber/whatever. What you don't want to do is have an unlicensed friend work on your home, and vice versa. There are no legal protections, and the i…
Extraordinary claims require extraordinary evidence.
It‘s hardly an extraordinary claim. Just because you can’t install a ceiling fan doesn’t mean it‘s an “extraordinary” feat that is “likely illegal”.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#184Earlier quoted context omitted.
Americans do not have electric kettles and need special circuits for electric clothes dryers.
Huh, what?! Mega TIL moment for me as an Australian with an electric kettle and dryer plugged into whatever power socket I wish! Reminds me of this great Technology Connections video: https://youtu.be/jMmUoZh3Hq4?si=3vSMHmU2ClwNRtow
I'm pretty sure in our current rental the kitchen and laundry are on the same circuit, which means I'll often have the dryer, washing machine, kettle, toaster and microwave drawing power at the same time. It's never been an issue.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#185Earlier quoted context omitted.
The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.
I’m hardly surprised, this is primarily a programming discussion website, and the highest voltage read on an average day here is in mV. It’s natural to be leery of things you have no experience in.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#186Re: Serving AI from the Basement – 192GB of VRAM Setup
#187Earlier quoted context omitted.
Extraordinary claims require extraordinary evidence.
As with most regulations in the "US" I have a feeling the answer is really something like "Depending on the city and state you live in the answer lies somewhere between 'go nuts' and 'that could lead to criminal charges and you being liable for everything that happens to the house and your neighbors kitchen sink'".
It's like that in Australia, liability and insurance hinge on licenced work by trade qualified professionals.
What is common here, in the handy crowd at least, is to do your own electrical, plumbing, gas work and leave it open and accessable for a licenced professional to check and sign off on.
You're still paying for an hour or two of their time and a surcharge for "taking on the responsibility" but it's often not an issue if the work is clean, to current code, and sanity tests correct (correct wiring, correct angles on plumbing, pressure testing on gas pipes).
Re: Serving AI from the Basement – 192GB of VRAM Setup
#188Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…
I tried self hosting LLM for commandline instant completion and guidance utility:
https://github.com/zerocorebeta/Option-K
But problem is even 7b models are too slow on my pc.
Hosted models are lightening fast. I considered possibility of buying hardware but decided against it.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#189Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…
The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
You can run a setup of 8x 4090 GPUs using 4x 1200W 240V power supplies (preferably HP HSTNS-PD30 Platinum Series), with a collective use of just around 20-amps, meaning it can easily run on a single 240V 20-amp breaker. This should be easily doable in a home where you typically have a 100 to 200A main power panel. Running 4x 1200W power supplies 24 hours a day will consume 115.2 kWh per day. At an electricity rate of $0.12 per kWh, this will cost approximately $13.82 per day or around $414.72 per month.
FYI, I can handle electrical system design and sheet metal enclosure design/fabrication for these rigs, but my software knowledge is limited when it comes to ML. If anyone's interested, I'd love to collaborate on a joint venture to produce these rigs commercially.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#190Earlier quoted context omitted.
What? No I just don’t know the difference, sorry. I am interested in learning more about running 405b parameter models, which I believe you can do on a 192gb M series Mac. The answer here is that the Nvidia system has much better performance. I’ve been focused on “can I even run the model” I didn’t think about the actual performance of the system.
You might enjoy last fortnight's The Register article: Buying a PC for local AI? These are the specs that matter https://www.theregister.com/2024/08/25/ai_pc_buying_guide/ It died without interest here: https://news.ycombinator.com/item?id=41347785 likely time of day, I'm mostly active in HN off peak hours.
Nice, thank you!