Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

251–260 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#251
post #167

Earlier quoted context omitted.

Extraordinary claims require extraordinary evidence.

It‘s hardly an extraordinary claim. Just because you can’t install a ceiling fan doesn’t mean it‘s an “extraordinary” feat that is “likely illegal”.

> insurance can’t deny the claim based solely on the fact that you performed the work yourself

_This_ is the claim that is extraordinary. I'm not saying that the government would bust down my door for doing work on my own home, but rather that the insurance company would then view that work as uninsured.

The entire business model of insurance agencies is to find new, creative, and unexpected ways to deny claims. That is how they make their money. To claim that they would accept liability for a property that's had uninspected work done by an unlicensed, untrained, unregistered individual is just that - extraordinary.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#252

Earlier quoted context omitted.

Worth mentioning - this also cuts the available bandwidth to each card by 50%.

While you're technically correct, assuming you're using PCIe 4.0 or higher, the performance difference between x8 and x16 is practically zero.

Is this because the graphics cards are not using all available PCIe bandwidth? Or why?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#253

Earlier quoted context omitted.

It's kinda hard to believe that someone would stumble onto the landmine of AI performance comparison between Apple Silicon and Nvidia hardware. People are going to be rude because this kinda behavior is genuinely indistinguishable from bad-faith trolling. From benchmarks alone, you can easily tell that the performance-per-watt of any Mac Studio gets annihilated by a 4090: https://browser.geekbench.com/opencl-benchmar…

> It's kinda hard to believe that someone would stumble onto the landmine of AI performance comparison between Apple Silicon and Nvidia hardware. I encourage you to update your beliefs about other people. I’m a very technical person, but I work in robotics closer to the hardware level - I design motor controllers and Linux motherboards and write firmware and platform level robotics stacks, but I’ve never done any wor…

It's simply bizarre that you would ask that question when the research to figure it all out is trivially accessed. Everyone thought that "unified memory" would be a boon when it was advertised, but Apple never delivered on a CUDA alternative. They killed OpenCL in the cradle, and pushed developers to use Metal Compute Shaders instead of a proper GPGPU layer. If you are an Apple dev, the mere existence of CoreML ought to be the white flag that makes you realize Apple hardware was never made for GPU compute.

Again, I'm not accusing you of bad-faith. I'm just saying that asking such a bald-faced and easily-Googled question is indistinguishable from flamebait. There is so much signalling that should suggest to you that Apple hardware is far from optimized for AI workloads. You can look at it from the software angle, where Apple has no accessible GPGPU primitives. You can look at it from a hardware perspective, where Apple cannot beat the performance-per-watt of desktop or datacenter Nvidia hardware. You can look at it from a practical perspective, where literally nobody is using Apple Silicon for cost-effective inference or training. Every single scrap of salient evidence suggests that Apple just doesn't care about AI and the industry cannot be bothered to do Apple's dirty work for them. Hell, even a passing familiarity with the existence of Xserve should say everything you need to know about Apple competing in markets they can't manipulate.

> funny in my circles people are talking about unions, AI compute exacerbating climate change, and AI being used to disenfranchise and make more precarious the tech working class.

Sounds like your circles aren't focused on technology, but popular culture and Twitter topics. Unionization, the "cost" of cloud and fictional AI-dominated futures were barely cutting-edge in the 90s, let alone today.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#254
post #235

Earlier quoted context omitted.

I don't think you are being fair to the previous poster. As I read it they are simply pointing out that there is precedent for such decentralized contribution of compute resources. However folding at home doesn't allow to reward users for their contributions AFAIK. So maybe if a Blockchain based reward system could be layered on top of that it could increase participation. That's a big if I grant you but don't see ho…

I think the word blockchain confuses people, including you and the previous poster. Maybe you could clarify your “layering” idea and how it would work for further discussion. Folding at home can track user contributions and issue micro/payments as they see fit. Crucially, this does not need an immutable chain of truth to do. Instead, if we added a blockchain, then we would require 2 sets of participants - those who r…

No idea about the previous poster but I'm pretty sure I know how blockchain(s) work thanks. I don't claim to have a concrete proposal but the idea of proof of useful work has been around for a while as a research area (https://eprint.iacr.org/2017/203.pdf). Having a system that supports arbitrary computations might be hard, but perhaps any task for which solutions are easy-to-verify but difficult to compute might be a good fit. Alternatively, if creating an open/decentralized compute system is a goal then a proof of stake blockchain could allow users to post tasks with associated rewards (again in cases where solutions are easy-to-verify but hard to compute).

Re: Serving AI from the Basement – 192GB of VRAM Setup

#255

Earlier quoted context omitted.

> Also, if you are at the point where you need to add a circut for power, you might need to seriously consider cooling, which could potentially be another side quest. There should be an easy/reliable way to channel "waste heat" from something like this to your hot water system. Actually, 4 or 5 kW continuous is a lot more than most domestic hot water services need. So in my usual manner of overcomplicating simple ide…

Let me know if you figure it out, I would be really interested hahaha

I'm doing this myself now. I have a homelab server setup and a hybrid water heater.

Stuffed the homelab next to the air intake of the water heater, now when I need hot water my water heater sucks the heat out of the air and puts it into the water.

It's obviously not 100% efficient, but at least it recaptures some of the waste heat and decreases my electrical bill somewhat.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#256
post #235

Earlier quoted context omitted.

I don't think you are being fair to the previous poster. As I read it they are simply pointing out that there is precedent for such decentralized contribution of compute resources. However folding at home doesn't allow to reward users for their contributions AFAIK. So maybe if a Blockchain based reward system could be layered on top of that it could increase participation. That's a big if I grant you but don't see ho…

I think the word blockchain confuses people, including you and the previous poster. Maybe you could clarify your “layering” idea and how it would work for further discussion. Folding at home can track user contributions and issue micro/payments as they see fit. Crucially, this does not need an immutable chain of truth to do. Instead, if we added a blockchain, then we would require 2 sets of participants - those who r…

I imagined if it was proof-of-work the mining would actually be the compute work requested. Everyone is racing to solve the problem just like in Bitcoin, except the problem is the requested GPU task. The fastest/first one to provide a result gets to update the ledger (and receives the reward).

Maybe you run a private platform too like git/GitHub if there are real world payments and user accounts, but I wonder why couldn't that technology be used? Does "blockchain" just have an irreparably bad name at this point?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#257
post #70

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

I can't believe a group of engineers are so afraid of residential power. It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

[deleted]

Re: Serving AI from the Basement – 192GB of VRAM Setup

#258

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

>Anything around ~2k watts on a single circuit breaker is likely to flip it I'm curious, how do you use e.g. a washing machine or an electric kettle, if 2kW is enough to flip your breaker? You should simply know your wiring limits. Breaker/wiring at my home won't even notice this.

I rent an old Victorian. I have one breaker line for the fridge and microwave and one line for basically everything else.

If that wasn’t the limit though, the fact that the machine is currently a space heater at 2 liquid cooled 4090’s would be.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#259

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

Oh yeah, my original setup was an RTX 4090 + an RTX 3090, and I swear one night I had the circuit breaker trip more than 15 times before I gave up. I have a UPS so I would run to the box before my system shuts down. Most houses are equipped with 15amp 120v breakers, these should never exceed 1500w, and their max is 1800w but then you're really risking it. So, as mentioned on the article, I actually have installed (2)…

Thanks! Yeah the 4090’s are very thirsty if you let them be, I haven’t played enough with throttling their voltage and how that affects perf. Looking forward to your articles.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#260

Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop…

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

Is suppose this is an American view. Most places with 240 you can run anything up to 3kW per socket most of the time. But you can also get a sparky and go for a cheap high current socket install on 240 or even pay a bit more to get 3 phase installed, if you have a valid enough use case.

Hell most kettles use 3kw. Tho for a big server I'd get it wired dedicated, same way power showers are done (7-12~ kW)

Post reply on HN