Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All You Need Is 4x 4090 GPUs to Train Your Own Model

sabareesh.com

Post reply on HN
Print view View original

71–80 of 125 posts

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#71

Post by Bancakes »

minimaxir wrote: post #10
_just7_ wrote: post #3

I would be much more intrested in a piece on what you can train with this kind of rig, rather than the rig itself

The bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks. Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.

I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#73

Post by angoragoats »

NKosmatos wrote: post #46

Wouldn’t a cluster of M4 minis cost less and provide more VRAM? There are posts about people getting decent performance for a lot less than 12k USD.

If you are willing and able to put together the type of system described in the OP (a workstation-class PC, with multiple discrete GPUs and often multiple power supplies), a Mac never makes sense. There are hardware options available at essentially every price point that beat (in some cases drastically) the performance and memory capacity of a Mac.

And I say this at the risk of being called pedantic, but a cluster of Mac minis would have zero VRAM.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#74

Post by tmostak »

minimaxir wrote: post #10
_just7_ wrote: post #3

I would be much more intrested in a piece on what you can train with this kind of rig, rather than the rig itself

The bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks. Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.

You should be able to train/full-fine-tune (i.e. full weight updates, not LoRA) a much larger model with 96GB of VRAM. I generally have been able to do a full fine-tune (which is equivalent to training a model from scratch) of 34B parameter models at full bf16 using 8XA100 servers (640GB of VRAM) if I enable gradient checkpointing, meaning a 96GB VRAM box should be able to handle models of up to 5B parameters. Of course if you use LoRA, you should be able to go much larger than this, depending on your rank.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#76

Post by FridgeSeal »

manmal wrote: post #41
gzer0 wrote: post #30

This is a great build, thanks for sharing your learnings. The best build I have seen so far had 6x4090's. Video: https://www.youtube.com/watch?v=C548PLVwjHA Specifications - GPU Accelerator - 6 x 24GB NVIDIA GeForce RTX 4090 - Processor - Intel Xeon W7-3465X, 28C/56T, 2.5GHz - 4.8GHz - Memory - 256GB (8x32GB) DDR5 ECC 4800MHz - System Drive - 2TB Samsung 980 PRO NVMe PCIe 4.0 M.2 SSD - Storage Drive - 4TB Samsung 870…

On Reddit there's reports of 8x4090, or even 8xH100. I don't know where people get this kind of money for this, and why they don't rent infra instead.

Probably because they are after a lot of fast, local storage, and _that_ is where rented ML infra providers will sting you.

Edit: could also just be more-money-than-sense. Never discount stupidity.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#79

Post by wruza »

jmward01 wrote: post #58

You can get 4060 ti 16GB cards for ~$450 or 4070 ti 16gb for ~850 instead of the $2.5k for a 4090. I wonder how well 4 of those cards would perform. The 4060 TDP is 165w instead of 450w for the 4090. The 4070 looks like the best tradeoff though for cost/power/etc though. You could probably set up an 8 card 4070 ti 16gb system for less than the 4 card 4090 system

I’ve heard that people buy multiple 24GB P40’s for a bucket of dirt. But that was for inference, not sure about training.

g Tesla p40 llm reddit

Post reply on HN