Zero-3 Offload: Scale DL models to trillion parameters without code changes
1–10 of 49 posts
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#2please hook this up to Jax!
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#3GPT-NeoX is an example project that is using deepspeed and Zero-3 offloading. The wider project intend to train a GPT-3 sized model and release it freely to the world.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#4GPT-NeoX is an example project that is using deepspeed and Zero-3 offloading. The wider project intend to train a GPT-3 sized model and release it freely to the world. https://github.com/EleutherAI/gpt-neox
It seems like Zero-3 doesn't work for them:
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#5Huggingface has been working on implementing this into their library, and it has some pretty amazing effects on the size of models you can train on a simple Colab.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#6Support for this was also added to [Fairscale](https://fairscale.readthedocs.io/en/latest/) and [Fairseq](https://github.com/pytorch/fairseq) last week. In particular, the Fairscale implementation can be used in any pyotrch project without requiring the use of the Deepspeed trainer.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#7This is also being added to pytorch
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#8How much data do you need to mitigate the risk of over fitting a trillion parameter model?
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#9GPT-NeoX is an example project that is using deepspeed and Zero-3 offloading. The wider project intend to train a GPT-3 sized model and release it freely to the world. https://github.com/EleutherAI/gpt-neox
It seems like Zero-3 doesn't work for them: https://github.com/EleutherAI/gpt-neox/issues/171
Did you even read through the issue? I don't see anything that indicates it won't work.
Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes
#10ELI5? All this techno babble just sounds like "it's faster because we optimized it". What are the nontrivial, new fundamental tricks?