Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to train large models on many GPUs? (2021)

lilianweng.github.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: How to train large models on many GPUs? (2021)

#21

Post by buildbot »

dauertewigkeit wrote: post #2

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over ho…

Any framework that "just works" tends to not work when some small change is needed or a new model with new data/compute roofline comes out.

Re: How to train large models on many GPUs? (2021)

#22

Post by TheGuyWhoCodes »

buildbot wrote: post #20
TheGuyWhoCodes wrote: post #19

Earlier quoted context omitted.

AMD Radeon Pro SSG had 4 nvme slots on the card itself but that was 2017 but with direct storage API that might be able to have some gains for large models.

I could never get a solid answer wether that was presented as memory to the GPU or just as a PCIE switch with NVME drives hanging off one side and the GPU on another.

As far as I remember it was presented as a drive and was good for sequential read but you had to use AMD's API to get the full benefit

Re: How to train large models on many GPUs? (2021)

#23

Post by thrtythreeforty »

amelius wrote: post #13

I'm waiting for GPU cards that allow the user to plug in memory modules.

The bandwidth and latency is as much a bottleneck as the capacity is. That's why recent ML and GPU chips have moved to on-package "high-bandwidth memory." Even if you were to add more high-bandwidth memory off-package (via lots of parallel DIMMs, say), it's difficult to actually manage this memory well, because of a host of competing factors, including a) on-chip resources (controllers etc) that must consume area and power to manage the new memory, b) increased design cost for validating handling the extra (crappier than HBM) memory, c) software not knowing how to hide the (much higher) latency of this memory - deep learning isn't as good at memory hierarchies as CPUs are, d) a chip that has spare on-chip bus bandwidth for this kind of thing has wasted bus bandwidth. Et cetera.

So if you are building GPUs or AI accelerators, you tend to just go ahead and build this in.

Re: How to train large models on many GPUs? (2021)

#24

Post by [deleted] »

thrtythreeforty wrote: post #23
amelius wrote: post #13

I'm waiting for GPU cards that allow the user to plug in memory modules.

The bandwidth and latency is as much a bottleneck as the capacity is. That's why recent ML and GPU chips have moved to on-package "high-bandwidth memory." Even if you were to add more high-bandwidth memory off-package (via lots of parallel DIMMs, say), it's difficult to actually manage this memory well, because of a host of competing factors, including a) on-chip resources (controllers etc) that must consume area and…

[deleted]

Re: How to train large models on many GPUs? (2021)

#25

Post by vvipgupta »

When training over multiple GPUs, it's hard not to think about Ray (https://docs.ray.io/en/latest/train/train.html). Ray, as an open-source project, has exploded over the last few years and helps with the memory bottleneck by segregating memory and computing.

FYI, I am not affiliated with Ray. However, I did write the following paper on scaling data-parallel training for large ML models ;) https://openreview.net/pdf?id=rygFWAEFwS

Also, another one of my papers talks about distributed training while reducing the communication bottleneck for distributed training: https://dl.acm.org/doi/pdf/10.1145/3447548.3467080

Re: How to train large models on many GPUs? (2021)

#26

Post by robertnishihara »

vvipgupta wrote: post #25

When training over multiple GPUs, it's hard not to think about Ray ( https://docs.ray.io/en/latest/train/train.html ). Ray, as an open-source project, has exploded over the last few years and helps with the memory bottleneck by segregating memory and computing. FYI, I am not affiliated with Ray. However, I did write the following paper on scaling data-parallel training for large ML models ;) https://openreview.net/pd…

I'm one of the Ray developers, thanks for the shoutout :)

If you're curious about how Ray is used for LLMs, here are some interesting examples of LLM projects using Ray!

- Alpa does training and serving with 175B parameter models https://github.com/alpa-projects/alpa

- GPT-J https://github.com/kingoflolz/mesh-transformer-jax

- Another HN thread on training LLMs with Ray (on TPUs in this case) https://news.ycombinator.com/item?id=27731168

- OpenAI fireside chat on the evolution of their infrastructure and usage of Ray for training https://www.youtube.com/watch?v=CqiL5QQnN64

- Cohere on their architecture for training LLMs https://www.youtube.com/watch?v=For8yLkZP5w&t=3s

Some other thoughts

1. There is a lot more we want to do to make Ray better for working with large language models and for making training, serving, and batch inference work well out of the box.

2. The original post is about training, but we actually see even more interest in fine-tuning and serving with LLMs, in part because there are good pre-trained models.

3. For LLMs, we see a lot of interest in Ray + Jax or Ray + TPUs relative to what we see in other use cases.

Re: How to train large models on many GPUs? (2021)

#27

Post by paulmd »

buildbot wrote: post #20
TheGuyWhoCodes wrote: post #19

Earlier quoted context omitted.

AMD Radeon Pro SSG had 4 nvme slots on the card itself but that was 2017 but with direct storage API that might be able to have some gains for large models.

I could never get a solid answer wether that was presented as memory to the GPU or just as a PCIE switch with NVME drives hanging off one side and the GPU on another.

PCIe switch with NVMe drives and the GPU behind it, they actually just are presented straight to windows even.

https://www.youtube.com/watch?v=-fEjoJO4lEM

In principle they could be used with an API like DirectStorage RDMA or CUDA GPUDirect RDMA (which dates back to Kepler) and in this case they would never need to talk to the CPU, given appropriate software support. But it's not going to be presented as GPU memory ever, it's going to work like a block storage device you can do RDMA requests against, most likely.

https://docs.nvidia.com/cuda/gpudirect-rdma/

https://developer.download.nvidia.com/video/gputechconf/gtc/...

Now technically - it all depends on what you mean by "as GPU memory" because PCIe is all RDMA anyway, even CPU-to-GPU is a RDMA operation. That's why there's the whole thing about "resizable BAR" etc - that's the aperture window in CPU memory that gets mapped in from the GPU memory.

So technically yes you can map those SSDs in "as GPU memory" via GPUDirect RDMA block storage (or DirectStorage), but you can do that with a regular NVMe SSD in an adapter card too. The SSG is just a "combo GPU+SSD card" in the same way QNAP makes those "combo network+SSD cards", but with a lot of fanfare/marketing around it.

To be absolutely fair, Fiji/Vega is a good design for that since it doesn't have a bunch of memory packages around it. But it wasn't what AMD trumpeted it as, as the LTT video describes, it was a very specific reaction to the question of 'workstation GPUs are using a lot of memory, HBM can't be scaled as high, how do we put more memory on a Fiji/Vega GPU for workstation users". And AMD's claims that HBM meant you could just swap everything around and not have to worry about framebuffer size were never true, the PCIe bus itself is not fast enough for that.

https://www.youtube.com/watch?v=NQ8YlXh-jbI&t=280s

Re: How to train large models on many GPUs? (2021)

#28

Post by my123 »

amelius wrote: post #13

I'm waiting for GPU cards that allow the user to plug in memory modules.

Not going to happen:

https://www.servethehome.com/wp-content/uploads/2022/08/NVID...

That unless an utterly revolutionary new interconnect technology comes...

LPDDR is getting adopted more and more too, with CPUs losing memory expansion capabilities in exchange of huge power savings.

Re: How to train large models on many GPUs? (2021)

#29

Post by amelius »

my123 wrote: post #28
amelius wrote: post #13

I'm waiting for GPU cards that allow the user to plug in memory modules.

Not going to happen: https://www.servethehome.com/wp-content/uploads/2022/08/NVID... That unless an utterly revolutionary new interconnect technology comes... LPDDR is getting adopted more and more too, with CPUs losing memory expansion capabilities in exchange of huge power savings.

I bet a lot of people would be happy if their GPU could train large models, even if it took 5x as long.

Re: How to train large models on many GPUs? (2021)

#30

Post by eternalban »

robertnishihara wrote: post #26
vvipgupta wrote: post #25

When training over multiple GPUs, it's hard not to think about Ray ( https://docs.ray.io/en/latest/train/train.html ). Ray, as an open-source project, has exploded over the last few years and helps with the memory bottleneck by segregating memory and computing. FYI, I am not affiliated with Ray. However, I did write the following paper on scaling data-parallel training for large ML models ;) https://openreview.net/pd…

I'm one of the Ray developers, thanks for the shoutout :) If you're curious about how Ray is used for LLMs, here are some interesting examples of LLM projects using Ray! - Alpa does training and serving with 175B parameter models https://github.com/alpa-projects/alpa - GPT-J https://github.com/kingoflolz/mesh-transformer-jax - Another HN thread on training LLMs with Ray (on TPUs in this case) https://news.ycombinator…

Do you see any convergence on wire (arrow?) and storage (pandas?) formats?
Post reply on HN