Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to train large models on many GPUs? (2021)

lilianweng.github.io

Post reply on HN
Print view View original

11–20 of 36 posts

Re: How to train large models on many GPUs? (2021)

#11

Post by sandkoan »

dauertewigkeit wrote: post #2

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…

Might this be what you're looking for: https://github.com/bigscience-workshop/petals ?

Re: How to train large models on many GPUs? (2021)

#12

Post by NerdyDrone »

guardiantesla wrote: post #9
dauertewigkeit wrote: post #2

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…

>Why isn't there a framework that does all this automatically for you? Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times https://www.mosaicml.com/blog/supercharge-training-composer Also worth checking out their “training from scratch” blog posts. Training StableDiffusion: https://www.mosaicml.com/blog/train…

Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer.

* It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train...

Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi...

I'm just a huge fan of their developer experience. It's up there with Transformers and Datasets as the nicest tools to use.

Re: How to train large models on many GPUs? (2021)

#14

Post by option »

dauertewigkeit wrote: post #2

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…

There is - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en...

Supports data, tensor, pipeline, sequence parallelisms, activation checkpointing, distributed optimizers, fused kernels and more.

Re: How to train large models on many GPUs? (2021)

#16

Post by amelius »

dauertewigkeit wrote: post #2

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…

> Why isn't there a framework that does all this automatically for you?

Question: could this be implemented in PyTorch in an opaque way? Or would it require changes to its API?

Re: How to train large models on many GPUs? (2021)

#17

Post by cma »

dauertewigkeit wrote: post #5
minimaxir wrote: post #4

Earlier quoted context omitted.

DeepSpeed became popular soon after this post was originally published and is natively supported by many PyTorch training frameworks. https://www.deepspeed.ai https://www.deepspeed.ai/training/

I tried that as well, but maybe I did not use it correctly. I did not see the full sharding that I was hoping for. I only saw results similiar to FSDP.

How about flexflow?

https://huggingface.co/transformers/v4.9.2/parallelism.html#...

Re: How to train large models on many GPUs? (2021)

#18

Post by atty »

dauertewigkeit wrote: post #2

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…

Beyond the other answers, I’ll point out that pytorch is developing tools that will make doing this work by hand or implementing in a framework much easier. They’re building a native DTensor implementation and testing out SPMD-style distributed models with pipelining. DTensor is in torch.distributed, and the SPMD code is in the repo called Tau under the pytorch org on github.

Re: How to train large models on many GPUs? (2021)

#20

Post by buildbot »

TheGuyWhoCodes wrote: post #19
amelius wrote: post #13

I'm waiting for GPU cards that allow the user to plug in memory modules.

AMD Radeon Pro SSG had 4 nvme slots on the card itself but that was 2017 but with direct storage API that might be able to have some gains for large models.

I could never get a solid answer wether that was presented as memory to the GPU or just as a PCIE switch with NVME drives hanging off one side and the GPU on another.
Post reply on HN