Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Run Llama 3 405B on Home Devices? Build AI Cluster

b4rtaz.medium.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#11

Post by imtringued »

adrian_b wrote: post #8
FloatArtifact wrote: post #4

An interesting idea, but there's no discussion about benchmarks in the article, cpu or other hardware requirements beyond 230 GB RAM for the cluster. It seems impractical that a home would have 4 machines with 64 gb ram that would be dedicated to a distributed system. Max, core count, 16 cores from Consumer AMD CPUs? From a cost perspective build a system with 256gb ram and AMD Epic CPU? The only thing I can think of…

Unless you buy some obsolete Epyc CPU with a small number of cores, which would be slower than even one up-to-date Ryzen 9 9950X (which can beat even 32-core older Epycs in compute-limited tasks that use vector instructions), 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000) and also faster than one box with an Epyc CPU and 256 GB, where the CPU alone c…

> 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000)

Yeah, but then the question arises: why? Assuming 4x $700 for a mini PC setup, you still only get like 360GB/s memory bandwidth. It is a lot of money with nothing to show for it, really. Your model will run at 2 tokens per second. A GPU setup within the same budget could run a 70B model at high speeds.

This is why I am kinda bullish on the XDNA NPU from AMD. It represents the low end. People who run small models below 16GB in size. Since the hardware will be available on almost every AMD laptop, it will be much easier to actually utilize in say a video game. As of today, it is kind of annoying when an LLM based game asks you for an API key. The out of the box experience isn't that good.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#12

Post by adrian_b »

imtringued wrote: post #11
adrian_b wrote: post #8

Earlier quoted context omitted.

Unless you buy some obsolete Epyc CPU with a small number of cores, which would be slower than even one up-to-date Ryzen 9 9950X (which can beat even 32-core older Epycs in compute-limited tasks that use vector instructions), 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000) and also faster than one box with an Epyc CPU and 256 GB, where the CPU alone c…

> 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000) Yeah, but then the question arises: why? Assuming 4x $700 for a mini PC setup, you still only get like 360GB/s memory bandwidth. It is a lot of money with nothing to show for it, really. Your model will run at 2 tokens per second. A GPU setup within the same budget could run a 70B model at high speeds.…

I have replied to the posting claiming that an Epyc might be better, which it is not.

The bandwidth per core is the same for a 16-core Ryzen and for a 96-core Epyc.

Of course if you compare it with a GPU, the GPU wins in memory bandwidth. However the GPUs are useless for FP64 computations, unless you are a big corporation that can afford the huge prices of the "datacenter" GPUs.

For ML/AI, GPUs are the normal choice, except when you cannot afford those with enough memory, when you may still want to fall back to cheap CPUs, as argued in the parent article.

A $1500 Ryzen 9 9950X system will have a double price in comparison with a mini-PC, but it will be 3 to 4 times faster in any program whose performance is dominated by the speed of the array operations. This speed ratio is true for the fastest CPU that could be put now in a mini-PC, i.e. AMD Strix Point (a HX 370 does 192 FP32 FMA per clock cycle, while a 9950X does 512 FP32 FMA per clock cycle and its clock frequency is much higher; it should be noted that these values are similar to those for integrated GPUs; the best iGPUs have a double number of ALUs, but their clock frequency is much lower, slightly more than half of that of a CPU). In comparison with a low-power Intel CPU or older AMD CPU, the speed advantage of 9950X would be even greater.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#13

Post by atemerev »

But this is not a problem, if you have money to buy 1TB memory, you can easily get a motherboard that supports it for around $800-$1000. It will be cheaper than attempting to build a cluster with 4x256GB RAM.

GPU inference is another thing, as high-VRAM GPUs are artificially priced way high so they could only be bought by corporations. However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB VRAM, you won’t have enough electricity to run it at home.

I am currently building a 4x4090 rig, but that’s probably the maximum I could have at home giving my budget / available power restrictions. And that’s only 96G VRAM, slightly more than a single A100.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#14

Post by RealStickman_ »

malux85 wrote: post #6
RealStickman_ wrote: post #5

Earlier quoted context omitted.

If you just want to play with this a bit, around 250$/month should give you enough metal when renting from cheap VDS/dedicated server providers.

Surely not with GPUs? Could you direct me to them please?

This project only supports CPU inference at the moment, so no, no GPUs. I just estimated what some 256GB or RAM might cost monthly on the low end.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#15

Post by RealStickman_ »

wkat4242 wrote: post #7
RealStickman_ wrote: post #5

Earlier quoted context omitted.

If you just want to play with this a bit, around 250$/month should give you enough metal when renting from cheap VDS/dedicated server providers.

250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it. I'd love to see a llama model that fits now economically inside 16GB. The 8b is a bit too small when quantised even to 8 bits. A 16-20b model would be perfect. But I think for 400b models to be viable, the hardware pricing really needs to catch up.

> 250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it.

Agreed, it's a lot of money and definitely more than I'd be willing to spend. However, compared to running such 400b models on a GPU cluster it's extremely cheap (and much slower)

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#16

Post by angoragoats »

atemerev wrote: post #13

But this is not a problem, if you have money to buy 1TB memory, you can easily get a motherboard that supports it for around $800-$1000. It will be cheaper than attempting to build a cluster with 4x256GB RAM. GPU inference is another thing, as high-VRAM GPUs are artificially priced way high so they could only be bought by corporations. However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB…

> However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB VRAM, you won’t have enough electricity to run it at home.

The average home (in the United States at least) has more than enough power to run 10 GPUs, or even more, with plenty left over for other appliances. Back of the napkin math is that ten high-end GPUs, at 350W each, need approximately 30-35A of power, and these days the bare minimum service in a home is 100A (and many homes have more than this; mine has 200A, for example). This could be accomplished on two standard 15A circuits, or perhaps more comfortably two 20A circuits.

You can also throttle your cards down to conserve power; I know that 3090s set to max power of 225W only lose ~5% of performance when running LLM workloads, while saving ~125W of power at full load.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#17

Post by atemerev »

angoragoats wrote: post #16
atemerev wrote: post #13

But this is not a problem, if you have money to buy 1TB memory, you can easily get a motherboard that supports it for around $800-$1000. It will be cheaper than attempting to build a cluster with 4x256GB RAM. GPU inference is another thing, as high-VRAM GPUs are artificially priced way high so they could only be bought by corporations. However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB…

> However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB VRAM, you won’t have enough electricity to run it at home. The average home (in the United States at least) has more than enough power to run 10 GPUs, or even more, with plenty left over for other appliances. Back of the napkin math is that ten high-end GPUs, at 350W each, need approximately 30-35A of power, and these days the bare mi…

Well, I live in an apartment in Switzerland, we have different standards.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#18

Post by angoragoats »

atemerev wrote: post #17
angoragoats wrote: post #16

Earlier quoted context omitted.

> However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB VRAM, you won’t have enough electricity to run it at home. The average home (in the United States at least) has more than enough power to run 10 GPUs, or even more, with plenty left over for other appliances. Back of the napkin math is that ten high-end GPUs, at 350W each, need approximately 30-35A of power, and these days the bare mi…

Well, I live in an apartment in Switzerland, we have different standards.

Yeah, makes sense. I was just trying to point out that "you won't have enough elecricity to run it at home" is not true for many single-family homeowners.

I'm not familiar with the typical Swiss apartment, so I'm sorry if this is a stupid question, but do you not think that your apartment could sustain a ~15A load at 230V? That's how much you'd need for ten RTX 4090s at full power.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#19

Post by atemerev »

angoragoats wrote: post #18
atemerev wrote: post #17

Earlier quoted context omitted.

Well, I live in an apartment in Switzerland, we have different standards.

Yeah, makes sense. I was just trying to point out that "you won't have enough elecricity to run it at home" is not true for many single-family homeowners. I'm not familiar with the typical Swiss apartment, so I'm sorry if this is a stupid question, but do you not think that your apartment could sustain a ~15A load at 230V? That's how much you'd need for ten RTX 4090s at full power.

It is not just the video cards. CPUs, mainboards, cooling systems, losses/inefficiencies… E.g. 4x4090 Comino Grando workstation is rated for 2811W, which is OK. 8x will probably trip my circuit breaker already, unless I plug it into higher-power circuit for kitchen appliances.

Re: How to Run Llama 3 405B on Home Devices? Build AI Cluster

#20

Post by angoragoats »

atemerev wrote: post #19
angoragoats wrote: post #18

Earlier quoted context omitted.

Yeah, makes sense. I was just trying to point out that "you won't have enough elecricity to run it at home" is not true for many single-family homeowners. I'm not familiar with the typical Swiss apartment, so I'm sorry if this is a stupid question, but do you not think that your apartment could sustain a ~15A load at 230V? That's how much you'd need for ten RTX 4090s at full power.

It is not just the video cards. CPUs, mainboards, cooling systems, losses/inefficiencies… E.g. 4x4090 Comino Grando workstation is rated for 2811W, which is OK. 8x will probably trip my circuit breaker already, unless I plug it into higher-power circuit for kitchen appliances.

Yep, in the US you’d need to use at least two separate circuits as well. I’m just saying that there probably is sufficient electricity for a crazy 10+ GPU setup, in most single family houses at least.
Post reply on HN