This is super impressive. I could not figure out for a while who exactly was running this project, but it looks like its Microsoft. Great work!
Zero-3 Offload: Scale DL models to trillion parameters without code changes
21–30 of 49 posts
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#22Earlier quoted context omitted.
Third paragraph or so in the overview: > ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency
Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.
If your pipeline uses only “classic” ml models, then this won’t make too much sense. It’s mostly applicable to NNs.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#23GPT-NeoX is an example project that is using deepspeed and Zero-3 offloading. The wider project intend to train a GPT-3 sized model and release it freely to the world. https://github.com/EleutherAI/gpt-neox
It seems like Zero-3 doesn't work for them: https://github.com/EleutherAI/gpt-neox/issues/171
Looks like they got it working recently https://github.com/EleutherAI/gpt-neox/pull/178
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#24Earlier quoted context omitted.
Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.
It doesn't sound like techno-babble to me. They've distributed storage across nodes rather than replicating on each node, hence the model size is now scalable with number of nodes rather than being limited to what could be stored on a single node.
But it's not clear how they managed to improve training on a single GPU: they say they can fit 40B model on a single V100.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#25Support for this was also added to [Fairscale]( https://fairscale.readthedocs.io/en/latest/ ) and [Fairseq]( https://github.com/pytorch/fairseq ) last week. In particular, the Fairscale implementation can be used in any pyotrch project without requiring the use of the Deepspeed trainer.
What are the relevant commits in Fairseq for this? I couldn't figure out the changes by looking at the commits from last week.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#26Earlier quoted context omitted.
It doesn't sound like techno-babble to me. They've distributed storage across nodes rather than replicating on each node, hence the model size is now scalable with number of nodes rather than being limited to what could be stored on a single node.
But it's not clear how they managed to improve training on a single GPU: they say they can fit 40B model on a single V100.
They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#27Earlier quoted context omitted.
Third paragraph or so in the overview: > ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency
Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.
It is mostly applicable to transformer models, the ideas in the paper would be alien if you work on computer vision.
In transformer models, big chunk of memory was parameters, and states for optimizers (because vanilla SGD not used there). The memory optimization technique that removes parameters duplication on each GPU or offload entirely to CPU makes sense.
In computer vision, big chunk of memory was hold by forward layer activations and the memory optimization technique applicable in these cases would be binomial checkpointing.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#28How much data do you need to mitigate the risk of over fitting a trillion parameter model?
You ideally need ~500GB of text, or so. EleutherAI's The Pile was designed to be just big enough to fit a 1t GPT efficiently, and you can get the various scaling curves out of the OA-related scaling papers. (You want the amount of data that fits into a single epoch, because if you reuse data, you get less bang for the FLOPs buck, and FLOPS constraints are right now much more binding than data or model size.)
This feels off by a couple of orders of magnitude, unless a significant number of the parameters are not independent.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#29ELI5? All this techno babble just sounds like "it's faster because we optimized it". What are the nontrivial, new fundamental tricks?
I think there is some explanation (on the previous model?) here, https://www.youtube.com/watch?v=tC01FRB0M7w
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#30See also zeroth order backpropagation which allows 300X faster training while not reducing throughput that much
https://arxiv.org/abs/2011.08895
How much zero-3 affect accuracy?