All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them
Also (at least in Southern California) electricity prices and how long the rig is on. Not as bad as the initial build cost, but run costs will add up over time.
All You Need Is 4x 4090 GPUs to Train Your Own Model
21–30 of 125 posts
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#22I would be much more intrested in a piece on what you can train with this kind of rig, rather than the rig itself
The bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks. Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.
Definitely agree but part of the reason why i built this to learn about all the overhead and gotchas
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#23All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them
You can get a used A100 for that cost and have better software support for training.
4090s are too small for training and you'll have to write your own suboptimal batching.
Unless you value the learning, it'd be better to rent GPUs in the cloud for training.
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#24All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them
You can get a used A100 for that cost and have better software support for training. 4090s are too small for training and you'll have to write your own suboptimal batching. Unless you value the learning, it'd be better to rent GPUs in the cloud for training.
Yup my initial reason behind is to learn all the quirks
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#25Earlier quoted context omitted.
You sure? In my experiments with multi gpu inference, I couldn't get anywhere close to max theoretical power draw.
Well during training all GPUs were consuming max ~ 450W
Thanks, good to know. Perhaps it is different for diffusion; with llms, layers are generally split across gpus, meaning inference has to happen on one gpu before the values can be passed between the layer split.
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#26This article was written or rewritten via your model right? The last paragraphs fell totally like AI. Anyway I'd like a follow up on the curating, cleaning and training part which is far more interesting than how to select hardware which we've been doing for over 25 years.
> Architecture Advantages: Enhanced ray tracing, Shader Execution Reordering, and DLSS 3 technology for improved efficiency.
This jumps right out as written by AI, these features have nothing to do with training LLMs.
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#27All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them
Also (at least in Southern California) electricity prices and how long the rig is on. Not as bad as the initial build cost, but run costs will add up over time.
That is real concern especially 4090 is not power efficient , as a100 and h100, h200. I live in Reno so it was ok
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#28Earlier quoted context omitted.
You can get a used A100 for that cost and have better software support for training. 4090s are too small for training and you'll have to write your own suboptimal batching. Unless you value the learning, it'd be better to rent GPUs in the cloud for training.
Yup my initial reason behind is to learn all the quirks
Consumer cards are a very different ecosystem, and you'll hit different use cases and challenges.
This might pull you down a path towards distilling and quantizing models, for instance.
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#29Interesting that DLSS 3 is mentioned as an advantage?
Because the article was clearly co-authored by AI
Re: All You Need Is 4x 4090 GPUs to Train Your Own Model
#30This is a great build, thanks for sharing your learnings.
The best build I have seen so far had 6x4090's. Video: https://www.youtube.com/watch?v=C548PLVwjHA
Specifications
- GPU Accelerator - 6 x 24GB NVIDIA GeForce RTX 4090
- Processor - Intel Xeon W7-3465X, 28C/56T, 2.5GHz - 4.8GHz
- Memory - 256GB (8x32GB) DDR5 ECC 4800MHz
- System Drive - 2TB Samsung 980 PRO NVMe PCIe 4.0 M.2 SSD
- Storage Drive - 4TB Samsung 870 EVO SSD
- Operating System - Ubuntu 20.04
The cost of this... not even sure. Astronomical.