MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
1–10 of 71 posts
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#2[dead]
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#3> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state
This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a lot.
I have a lot more CPU RAM in my PC, and this would likely increase the size of models I can train locally.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#4This would likely only get used for small finetuning jobs. It’s too slow for the scale of pretraining.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#5This would likely only get used for small finetuning jobs. It’s too slow for the scale of pretraining.
[deleted]
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#6> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a…
[deleted]
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#7Seems similar to Microsoft DeepSpeed.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#8I was wondering how well this would work :) You can definitely push this further, the question is: how well can the gradients and updates compress?
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#9This would likely only get used for small finetuning jobs. It’s too slow for the scale of pretraining.
It’s too slow for the scale of pretraining.
There isn't really such a thing as 'too slow' as an objective fact though. It depends on how much patience and money for electricity you have. In AI image gen circles I see people complaining if a model takes more than 5s to generate an image, and other people on very limited hardware who happily wait half an hour per image. It's hard to make a judgement call about what 'too slow' means. It's quite subjective.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#10This would likely only get used for small finetuning jobs. It’s too slow for the scale of pretraining.
It’s too slow for the scale of pretraining. There isn't really such a thing as 'too slow' as an objective fact though. It depends on how much patience and money for electricity you have. In AI image gen circles I see people complaining if a model takes more than 5s to generate an image, and other people on very limited hardware who happily wait half an hour per image. It's hard to make a judgement call about what 'to…
If it would take so long to train that the model will be obsolete before the training is finished that might be considered too long. With ML you can definitely hit a point where it is too slow for any practical purpose.