Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to train large models on many GPUs? (2021)

lilianweng.github.io

Post reply on HN
Print view View original

31–36 of 36 posts

Re: How to train large models on many GPUs? (2021)

#31

Post by kanyethegreat »

NerdyDrone wrote: post #12
guardiantesla wrote: post #9

Earlier quoted context omitted.

>Why isn't there a framework that does all this automatically for you? Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times https://www.mosaicml.com/blog/supercharge-training-composer Also worth checking out their “training from scratch” blog posts. Training StableDiffusion: https://www.mosaicml.com/blog/train…

Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer . * It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train... Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi... I'm just a huge fan of…

> Transformers and Datasets

Are these specific libraries?

Re: How to train large models on many GPUs? (2021)

#32

Post by pavelstoev »

robertnishihara wrote: post #26
vvipgupta wrote: post #25

When training over multiple GPUs, it's hard not to think about Ray ( https://docs.ray.io/en/latest/train/train.html ). Ray, as an open-source project, has exploded over the last few years and helps with the memory bottleneck by segregating memory and computing. FYI, I am not affiliated with Ray. However, I did write the following paper on scaling data-parallel training for large ML models ;) https://openreview.net/pd…

I'm one of the Ray developers, thanks for the shoutout :) If you're curious about how Ray is used for LLMs, here are some interesting examples of LLM projects using Ray! - Alpa does training and serving with 175B parameter models https://github.com/alpa-projects/alpa - GPT-J https://github.com/kingoflolz/mesh-transformer-jax - Another HN thread on training LLMs with Ray (on TPUs in this case) https://news.ycombinator…

And we can make Ray more efficient by optimizing GPU hardware utilization https://centml.ai/

Re: How to train large models on many GPUs? (2021)

#33

Post by my123 »

amelius wrote: post #29
my123 wrote: post #28

Earlier quoted context omitted.

Not going to happen: https://www.servethehome.com/wp-content/uploads/2022/08/NVID... That unless an utterly revolutionary new interconnect technology comes... LPDDR is getting adopted more and more too, with CPUs losing memory expansion capabilities in exchange of huge power savings.

I bet a lot of people would be happy if their GPU could train large models, even if it took 5x as long.

That's getting a multi-level DRAM config, see Grace Superchip for an example.

Or just a plain slower GPU: Apple's GPU series, but that's at very much higher price tags than desktop GPUs at a given perf level.

Re: How to train large models on many GPUs? (2021)

#34

Post by swyx »

kanyethegreat wrote: post #31
NerdyDrone wrote: post #12

Earlier quoted context omitted.

Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer . * It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train... Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi... I'm just a huge fan of…

> Transformers and Datasets Are these specific libraries?

https://github.com/huggingface/datasets

https://github.com/huggingface/transformers

Re: How to train large models on many GPUs? (2021)

#35

Post by q1w2 »

robertnishihara wrote: post #26
vvipgupta wrote: post #25

When training over multiple GPUs, it's hard not to think about Ray ( https://docs.ray.io/en/latest/train/train.html ). Ray, as an open-source project, has exploded over the last few years and helps with the memory bottleneck by segregating memory and computing. FYI, I am not affiliated with Ray. However, I did write the following paper on scaling data-parallel training for large ML models ;) https://openreview.net/pd…

I'm one of the Ray developers, thanks for the shoutout :) If you're curious about how Ray is used for LLMs, here are some interesting examples of LLM projects using Ray! - Alpa does training and serving with 175B parameter models https://github.com/alpa-projects/alpa - GPT-J https://github.com/kingoflolz/mesh-transformer-jax - Another HN thread on training LLMs with Ray (on TPUs in this case) https://news.ycombinator…

Will it work with a PC that has 7 AMD Vega GPUs?

Re: How to train large models on many GPUs? (2021)

#36

Post by robertnishihara »

q1w2 wrote: post #35
robertnishihara wrote: post #26

Earlier quoted context omitted.

I'm one of the Ray developers, thanks for the shoutout :) If you're curious about how Ray is used for LLMs, here are some interesting examples of LLM projects using Ray! - Alpa does training and serving with 175B parameter models https://github.com/alpa-projects/alpa - GPT-J https://github.com/kingoflolz/mesh-transformer-jax - Another HN thread on training LLMs with Ray (on TPUs in this case) https://news.ycombinator…

Will it work with a PC that has 7 AMD Vega GPUs?

Yes, but this will largely come down to whether the deep learning framework that you're using (PyTorch, TensorFlow, Jax, etc) works well in that setting. Ray is pretty framework and hardware agnostic and can be used to schedule / scale different ML frameworks on different types of devices (CPUs, GPUs, TPUs, etc), but the actual logic for running code on the accelerators lives in the deep learning framework.
Post reply on HN