How to train large models on many GPUs? (2021)
lilianweng.github.io
How to train large models on many GPUs? (2021)
1–10 of 36 posts
Re: How to train large models on many GPUs? (2021)
#2Why isn't there a framework that does all this automatically for you?
I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU.
I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over how many GPUs manually.
I would be interested if anyone knows of a framework that manages this automatically and just works.
EDIT: BTW I am talking about model sharding not data parallelism which works very well with DDP.
Re: How to train large models on many GPUs? (2021)
#3Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…
It's a fair question.
Nvidia's NCCL and AMD's RCCL provide parallelism constructs that really are hidden at the framework level (such as PyT).
However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact.
At least in scientific computing, I've never observed effective means of automatic parallelism expressed across many nodes despite decades of research. I'm not optimistic this will be effective anytime soon.
Re: How to train large models on many GPUs? (2021)
#4Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…
DeepSpeed became popular soon after this post was originally published and is natively supported by many PyTorch training frameworks.
Re: How to train large models on many GPUs? (2021)
#5Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…
DeepSpeed became popular soon after this post was originally published and is natively supported by many PyTorch training frameworks. https://www.deepspeed.ai https://www.deepspeed.ai/training/
I tried that as well, but maybe I did not use it correctly.
I did not see the full sharding that I was hoping for.
I only saw results similiar to FSDP.
Re: How to train large models on many GPUs? (2021)
#6[flagged]
Re: How to train large models on many GPUs? (2021)
#7[flagged]
Re: How to train large models on many GPUs? (2021)
#8Discussed (a bit) at the time:
How to train large models on many GPUs? - https://news.ycombinator.com/item?id=28657797 - Sept 2021 (9 comments)
Re: How to train large models on many GPUs? (2021)
#9Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…
>Why isn't there a framework that does all this automatically for you?
Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times
https://www.mosaicml.com/blog/supercharge-training-composer
Also worth checking out their “training from scratch” blog posts.
Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from...
Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training...
Re: How to train large models on many GPUs? (2021)
#10Discussed (a bit) at the time: How to train large models on many GPUs? - https://news.ycombinator.com/item?id=28657797 - Sept 2021 (9 comments)
Somewhat amazed, dang, that this topic is not discussed more widely here or elsewhere. There is a lot of HPC and DS expertise out there which lacks understanding of ML system architecture (in the sense of the deployed machinery in toto).
Her follow up post [1] is also recommended for those who (like me, are experienced but not in ML) finally had things click because of the OP writeup:
Large Transformer Model Inference Optimization (2023)
https://lilianweng.github.io/posts/2023-01-10-inference-opti...
A very cool cite from that article is LLM.int8(): https://arxiv.org/abs/2208.07339