Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Run Llama 3 405B on Home Devices? Build AI Cluster

b4rtaz.medium.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#21

Post by BizarroLand »

angoragoats wrote: post #20
atemerev wrote: post #19

Earlier quoted context omitted.

It is not just the video cards. CPUs, mainboards, cooling systems, losses/inefficiencies… E.g. 4x4090 Comino Grando workstation is rated for 2811W, which is OK. 8x will probably trip my circuit breaker already, unless I plug it into higher-power circuit for kitchen appliances.

Yep, in the US you’d need to use at least two separate circuits as well. I’m just saying that there probably is sufficient electricity for a crazy 10+ GPU setup, in most single family houses at least.

If you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#22

Post by angoragoats »

BizarroLand wrote: post #21
angoragoats wrote: post #20

Earlier quoted context omitted.

Yep, in the US you’d need to use at least two separate circuits as well. I’m just saying that there probably is sufficient electricity for a crazy 10+ GPU setup, in most single family houses at least.

If you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.

I don’t condone doing this under any circumstances without first consulting an electrician. Lots of things could go wrong; for example, 40A @ 120V is not a very common configuration, and most standard household receptacles are rated for only 20A at most. Consult an electrician, folks.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#23

Post by Kirby64 »

BizarroLand wrote: post #21
angoragoats wrote: post #20

Earlier quoted context omitted.

Yep, in the US you’d need to use at least two separate circuits as well. I’m just saying that there probably is sufficient electricity for a crazy 10+ GPU setup, in most single family houses at least.

If you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.

Or just run your PSU off 240V. Much more standard supply, and most switched mode supplies already support 240V. They’re typically more efficient, too.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#24

Post by [deleted] »

angoragoats wrote: post #22
BizarroLand wrote: post #21

Earlier quoted context omitted.

If you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.

I don’t condone doing this under any circumstances without first consulting an electrician. Lots of things could go wrong; for example, 40A @ 120V is not a very common configuration, and most standard household receptacles are rated for only 20A at most. Consult an electrician, folks.

[deleted]

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#25

Post by BizarroLand »

angoragoats wrote: post #22
BizarroLand wrote: post #21

Earlier quoted context omitted.

If you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.

I don’t condone doing this under any circumstances without first consulting an electrician. Lots of things could go wrong; for example, 40A @ 120V is not a very common configuration, and most standard household receptacles are rated for only 20A at most. Consult an electrician, folks.

Thanks, I did mean to imply that you would be paying an electrician to do the work to code, but I did also overlook that someone who wants to run a 405b Llama ai at home might be more of a diy'er than the average person.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#26

Post by angoragoats »

Kirby64 wrote: post #23
BizarroLand wrote: post #21

Earlier quoted context omitted.

If you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.

Or just run your PSU off 240V. Much more standard supply, and most switched mode supplies already support 240V. They’re typically more efficient, too.

That works for sure, though it's less common in the US to have a 240v circuit available (with an appropriate receptacle) anywhere you might put a large server. Using two 120v circuits, you're more likely to be able to set it up with what you have now, without requiring any electrical work.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#27

Post by Kirby64 »

angoragoats wrote: post #26
Kirby64 wrote: post #23

Earlier quoted context omitted.

Or just run your PSU off 240V. Much more standard supply, and most switched mode supplies already support 240V. They’re typically more efficient, too.

That works for sure, though it's less common in the US to have a 240v circuit available (with an appropriate receptacle) anywhere you might put a large server. Using two 120v circuits, you're more likely to be able to set it up with what you have now, without requiring any electrical work.

The comparison was a 120V 40A circuit. A 240V-20A socket would be more common than a 120V 50A socket. Both technically exist, but a 240V 20A outlet would be cheaper, safer, and more common. A 240V 20A outlet is actually somewhat common for larger window mounted AC units.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#28

Post by angoragoats »

Kirby64 wrote: post #27
angoragoats wrote: post #26

Earlier quoted context omitted.

That works for sure, though it's less common in the US to have a 240v circuit available (with an appropriate receptacle) anywhere you might put a large server. Using two 120v circuits, you're more likely to be able to set it up with what you have now, without requiring any electrical work.

The comparison was a 120V 40A circuit. A 240V-20A socket would be more common than a 120V 50A socket. Both technically exist, but a 240V 20A outlet would be cheaper, safer, and more common. A 240V 20A outlet is actually somewhat common for larger window mounted AC units.

> The comparison was a 120V 40A circuit.

Yes, which was a suggestion in reply to my comment mentioning the use of two standard 120v circuits, which I still maintain is the easiest and cheapest path for most people.

Post reply on HN