Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-3 Offload: Scale DL models to trillion parameters without code changes

deepspeed.ai

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#11

Post by jiofih »

dataangel wrote: post #10

ELI5? All this techno babble just sounds like "it's faster because we optimized it". What are the nontrivial, new fundamental tricks?

Third paragraph or so in the overview:

> ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#12

Post by dataangel »

jiofih wrote: post #11
dataangel wrote: post #10

ELI5? All this techno babble just sounds like "it's faster because we optimized it". What are the nontrivial, new fundamental tricks?

Third paragraph or so in the overview: > ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency

Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#13

Post by ansk »

Question for someone knowledgable about this: if I have a model which is large -- but small enough that I can fit a single training example on GPU -- does this approach offer speedups compared to simple gradient accumulation? Or is this only useful for models which are so large that the model parameters themselves are overwhelming GPU memory?

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#14

Post by ma2rten »

dqpb wrote: post #9
ma2rten wrote: post #4

Earlier quoted context omitted.

It seems like Zero-3 doesn't work for them: https://github.com/EleutherAI/gpt-neox/issues/171

Did you even read through the issue? I don't see anything that indicates it won't work.

Yes, I did. The last comment is a traceback and an explanation what would have to be done to fix it.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#15

Post by vladf »

Alternatively, one could get rid of the memory used by optimizers entirely by switching to vanilla SGD.

I haven’t tried this on transformers and maybe that’s what breaks down here but in “classic” supervised settings I’ve found SGD with schedule tuning just as fast as Adam.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#16

Post by cambalache »

dataangel wrote: post #12
jiofih wrote: post #11

Earlier quoted context omitted.

Third paragraph or so in the overview: > ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency

Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.

The product is obviously not for you but for clueless PHBs who want the "latest and best" for the team so those useless ML engineers can finally put his brilliant idea in production with a less than 1% prediction error.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#17

Post by zachthewf »

dataangel wrote: post #12
jiofih wrote: post #11

Earlier quoted context omitted.

Third paragraph or so in the overview: > ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency

Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.

It doesn't sound like techno-babble to me. They've distributed storage across nodes rather than replicating on each node, hence the model size is now scalable with number of nodes rather than being limited to what could be stored on a single node.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#18

Post by gwern »

vladf wrote: post #15

Alternatively, one could get rid of the memory used by optimizers entirely by switching to vanilla SGD. I haven’t tried this on transformers and maybe that’s what breaks down here but in “classic” supervised settings I’ve found SGD with schedule tuning just as fast as Adam.

SGD doesn't work on large Transformers, no. You need something like AdamW.

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#19

Post by gwern »

andrewprock wrote: post #8

How much data do you need to mitigate the risk of over fitting a trillion parameter model?

You ideally need ~500GB of text, or so. EleutherAI's The Pile was designed to be just big enough to fit a 1t GPT efficiently, and you can get the various scaling curves out of the OA-related scaling papers. (You want the amount of data that fits into a single epoch, because if you reuse data, you get less bang for the FLOPs buck, and FLOPS constraints are right now much more binding than data or model size.)

Re: Zero-3 Offload: Scale DL models to trillion parameters without code changes

#20

Post by jiofih »

dataangel wrote: post #12
jiofih wrote: post #11

Earlier quoted context omitted.

Third paragraph or so in the overview: > ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency

Yeah that would be the techno-babble. I've been working on a machine learning pipeline for 6 years and I still have no idea what this means.

You can read the paper here: https://arxiv.org/abs/1910.02054
Post reply on HN