Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-3 Offload: Scale DL models to trillion parameters without code changes

deepspeed.ai

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#41

Post by andrewprock »

gwern wrote: post #32
andrewprock wrote: post #28

Earlier quoted context omitted.

This feels off by a couple of orders of magnitude, unless a significant number of the parameters are not independent.

It's quite amusing. The standard statistical theory does not work at all in estimating data vs model size, and the bounds are all vacuously large. It's a very active area of research, understanding why models act so simple when overparameterized and coming up with real measures of model complexity. Lots to read there if you are interested in such things.

That just means that the parameters are not independent.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#42

Post by BadInformatics »

leogao wrote: post #40
dataangel wrote: post #12

Earlier quoted context omitted.

Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.

I have a reasonable amount of experience with distributed machine learning (and transformers in particular, too) and I have to 100% agree that this blog post (and even the ZeRO paper) is largely technobabble. I don't doubt that this might really work, but how it works is not elucidated very well, and I'm still not 100% sure I understand what they actually did. For anyone who still thinks the blog post has substance:…

Having followed this DeepSpeed stuff for a little while, the ZeRO paper is probably as close as you can get to an ELI5 because there's no singular brilliant idea behind this. Most of the ideas have been explored already (see e.g. the PyTorch DDP paper), but ZeRO takes them to their logical conclusion by throwing a TON of engineering work into the equation. For example, they implement custom fused kernels on CPU/GPU and a hand-vectorized Adam implementation.

I found that this earlier blog post [2] has a much better deep dive (with decent animations and more) into the underlying architecture. The ZeRO-Offload paper [3] also has far more detail about that part of the pipeline.

[1] https://arxiv.org/abs/2006.15704 [2] https://www.microsoft.com/en-us/research/blog/deepspeed-extr... [3] https://arxiv.org/abs/2101.06840

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#43

Post by leogao »

BadInformatics wrote: post #42
leogao wrote: post #40

Earlier quoted context omitted.

I have a reasonable amount of experience with distributed machine learning (and transformers in particular, too) and I have to 100% agree that this blog post (and even the ZeRO paper) is largely technobabble. I don't doubt that this might really work, but how it works is not elucidated very well, and I'm still not 100% sure I understand what they actually did. For anyone who still thinks the blog post has substance:…

Having followed this DeepSpeed stuff for a little while, the ZeRO paper is probably as close as you can get to an ELI5 because there's no singular brilliant idea behind this. Most of the ideas have been explored already (see e.g. the PyTorch DDP paper), but ZeRO takes them to their logical conclusion by throwing a TON of engineering work into the equation. For example, they implement custom fused kernels on CPU/GPU a…

My impression from reading the paper is most of the other optimizations (custom kernels, contiguous memory, checkpointing, etc) are orthogonal to the partitioning stuff. That seems to imply that ZeRO is model+pipeline parallel plus a bunch of miscellaneous bits. But they seem to emphasize that this isn't what their partitioning is, and that's the part that perplexes me the most. To be specific, I'd like someone to explain how their magical zero-redundancy data parallel (termed ZeRO-DP in the paper) works and how it's different from from model+pipeline parallel, and their paper is awfully sparse on that.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#44

Post by gwern »

andrewprock wrote: post #41
gwern wrote: post #32

Earlier quoted context omitted.

It's quite amusing. The standard statistical theory does not work at all in estimating data vs model size, and the bounds are all vacuously large. It's a very active area of research, understanding why models act so simple when overparameterized and coming up with real measures of model complexity. Lots to read there if you are interested in such things.

That just means that the parameters are not independent.

But you can fit randomly-generated labels!

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#45

Post by stephenroller »

diptanu wrote: post #25
stephenroller wrote: post #6

Support for this was also added to [Fairscale]( https://fairscale.readthedocs.io/en/latest/ ) and [Fairseq]( https://github.com/pytorch/fairseq ) last week. In particular, the Fairscale implementation can be used in any pyotrch project without requiring the use of the Deepspeed trainer.

What are the relevant commits in Fairseq for this? I couldn't figure out the changes by looking at the commits from last week.

https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#46

Post by ma2rten »

minimaxir wrote: post #31
ma2rten wrote: post #14

Earlier quoted context omitted.

Yes, I did. The last comment is a traceback and an explanation what would have to be done to fix it.

Your comment implied it's not possible at all for them to use it, not that it's currently not working.

I guess an ambiguity in the English language is at fault here. I meant it literally doesn't work for them, and not the idiom.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#47

Post by BadInformatics »

leogao wrote: post #43
BadInformatics wrote: post #42

Earlier quoted context omitted.

Having followed this DeepSpeed stuff for a little while, the ZeRO paper is probably as close as you can get to an ELI5 because there's no singular brilliant idea behind this. Most of the ideas have been explored already (see e.g. the PyTorch DDP paper), but ZeRO takes them to their logical conclusion by throwing a TON of engineering work into the equation. For example, they implement custom fused kernels on CPU/GPU a…

My impression from reading the paper is most of the other optimizations (custom kernels, contiguous memory, checkpointing, etc) are orthogonal to the partitioning stuff. That seems to imply that ZeRO is model+pipeline parallel plus a bunch of miscellaneous bits. But they seem to emphasize that this isn't what their partitioning is, and that's the part that perplexes me the most. To be specific, I'd like someone to ex…

> My impression from reading the paper is most of the other optimizations (custom kernels, contiguous memory, checkpointing, etc) are orthogonal to the partitioning stuff

This is true, I include them as examples of the amount of engineering work involved because using the partitioning as an example would require recapitulating their blog post :)

> But they seem to emphasize that this isn't what their partitioning is, and that's the part that perplexes me the most. To be specific, I'd like someone to explain how their magical zero-redundancy data parallel (termed ZeRO-DP in the paper) works and how it's different from from model+pipeline parallel, and their paper is awfully sparse on that.

Again, https://www.microsoft.com/en-us/research/blog/deepspeed-extr... is a much better resource on this. There really isn't any magic going on, nor are many of these ideas (checkpointing, model state sharding, bucketing, JIT communication of new states interleaved with compute, etc.) new when considered in isolation. ZeRO is data + model + pipeline parallel, but optimized to the nines and actually usable as a production library.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#48

Post by dron57 »

The_rationalist wrote: post #37
gwern wrote: post #18

Earlier quoted context omitted.

SGD doesn't work on large Transformers, no. You need something like AdamW.

Mish is generally superior to RadamW https://lessw.medium.com/meet-mish-new-state-of-the-art-ai-a...

That's cool, but Mish is an activation function while SGD and AdamW are optimizers. Apples and oranges.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#49

Post by andrewprock »

gwern wrote: post #44
andrewprock wrote: post #41

Earlier quoted context omitted.

That just means that the parameters are not independent.

But you can fit randomly-generated labels!

That's not in any way surprising. When you have more parameters than data, this is trivial.
Post reply on HN