Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All You Need Is 4x 4090 GPUs to Train Your Own Model

sabareesh.com

Post reply on HN
Print view View original

41–50 of 125 posts

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#41

Post by manmal »

gzer0 wrote: post #30

This is a great build, thanks for sharing your learnings. The best build I have seen so far had 6x4090's. Video: https://www.youtube.com/watch?v=C548PLVwjHA Specifications - GPU Accelerator - 6 x 24GB NVIDIA GeForce RTX 4090 - Processor - Intel Xeon W7-3465X, 28C/56T, 2.5GHz - 4.8GHz - Memory - 256GB (8x32GB) DDR5 ECC 4800MHz - System Drive - 2TB Samsung 980 PRO NVMe PCIe 4.0 M.2 SSD - Storage Drive - 4TB Samsung 870…

On Reddit there's reports of 8x4090, or even 8xH100. I don't know where people get this kind of money for this, and why they don't rent infra instead.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#42

Post by belter »

gzer0 wrote: post #30

This is a great build, thanks for sharing your learnings. The best build I have seen so far had 6x4090's. Video: https://www.youtube.com/watch?v=C548PLVwjHA Specifications - GPU Accelerator - 6 x 24GB NVIDIA GeForce RTX 4090 - Processor - Intel Xeon W7-3465X, 28C/56T, 2.5GHz - 4.8GHz - Memory - 256GB (8x32GB) DDR5 ECC 4800MHz - System Drive - 2TB Samsung 980 PRO NVMe PCIe 4.0 M.2 SSD - Storage Drive - 4TB Samsung 870…

Don't forget to talk to your local power company one year in advance. They will need to upgrade your local substation transformer... :-)

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#43

Post by [deleted] »

kristopolous wrote: post #36
master_crab wrote: post #4

All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them

The GPU rental market is fairly reasonable. There's lots of companies doing it. (I work at one of them). 4x 4090 can be fetched for around $0.40/hour on some platforms ... about $1.20 on others depending on how available you want it. Regardless, all in, you can do an average 10-or-so-day train for If you want on-prem, wait a few months. The supply of 5000 series (probably announced at CES in a few days) should push m…

[deleted]

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#44

Post by whimsicalism »

kristopolous wrote: post #36
master_crab wrote: post #4

All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them

The GPU rental market is fairly reasonable. There's lots of companies doing it. (I work at one of them). 4x 4090 can be fetched for around $0.40/hour on some platforms ... about $1.20 on others depending on how available you want it. Regardless, all in, you can do an average 10-or-so-day train for If you want on-prem, wait a few months. The supply of 5000 series (probably announced at CES in a few days) should push m…

don't you need nvlink? feel like an 80gb a100 would start being worth it at a $1.20/4x 4090 price point

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#45

Post by leoc »

bb88 wrote: post #32
fzzzy wrote: post #13

Earlier quoted context omitted.

You sure? In my experiments with multi gpu inference, I couldn't get anywhere close to max theoretical power draw.

Yes! His power supplies are 2x1500 Watt. That puts it at 3KW max which is more than a 20A circuit can provide (2400W). The standard outlet is typically rated at 15 amps or 1800W. And the 15A breaker is on one circuit. You can get 20A circuits but they need to be wired for it, and replacing the breaker won't cut it. Assuming his GPU is ~450W (his number) and power supplies are 80% efficient, well that means he's pulli…

In the US. The UK or EU will do you 3000W out of a standard domestic socket.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#49

Post by fzzzy »

Y_Y wrote: post #31
fzzzy wrote: post #25

Earlier quoted context omitted.

Thanks, good to know. Perhaps it is different for diffusion; with llms, layers are generally split across gpus, meaning inference has to happen on one gpu before the values can be passed between the layer split.

That's only if your model is too big for a single GPU and you're not batching.

Yes, that's what I was doing. Thanks for the info.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#50

Post by KeplerBoy »

Let's talk riser cables. I keep encountering issues with riser connectors claiming to support PCIe 4.0, which seem to have sub-par performance. They work fine with the GPUs and NICs I tested them with, but attaching a nvme drive causes all kinds of issues and prevents the machine from booting. I guess nvme isn't as tolerant of elevated bit-error-rates.

That just doesn't inspire a lot of confidence in those risers, so now I'm contemplating mcio risers.

Post reply on HN