Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GShard: Scaling giant models with conditional computation and automatic sharding

arxiv.org

Post reply on HN
Print view View original

31–37 of 37 posts

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#31

Post by modeless »

dna_polymerase wrote: post #28
modeless wrote: post #24

The most important advancements in machine learning for the next 10 years at least will be in hardware, and the software to take advantage of said hardware. You could even say that was already true starting with AlexNet, but it's even more obvious now with these enormous models. We've barely scratched the surface of what's possible. Even if Moore's Law was dead (though it seems that TSMC may keep it alive for a bit l…

Hardware will be a huge part, yes, but algorithmic advances would be even better. Utilizing existing commodity hardware to full extent is where the money is at. Specialized hardware will probably remain just that, specialized and mostly too expensive.

There will be algorithmic advances, but they will benefit larger models too. Larger models will still win. The value of having the best AI is so great that it will be worth nearly any level of investment in hardware to the large tech companies that can afford it.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#32

Post by gwern »

cs702 wrote: post #29
ma2rten wrote: post #25

Earlier quoted context omitted.

This is a sparse model. You can't just compare the parameter count against dense models like GPT-3. Otherwise, Google already had a 137B parameter model in 2017: https://arxiv.org/abs/1701.06538

The model is sparsely-gated, not sparse. The individual experts in each mixture of experts are dense layers but they're sparsely activated, i.e., on each forward pass only some of them are conditionally used. As to comparing parameter counts, I disagree with you. I think it's perfectly OK to compare parameter counts for different kinds of models. It would also be perfectly OK to compare, say, computational efficiency…

Probably the most relevant comparison here would be a mix of wallclock-hours and FLOPS. The MoE may be inefficient on a parameter level, but it may be the most efficient way to convert FLOPS into model power (sort of like how you currently do better making models wider than deeper - experts are the ultimate 'width').

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#33

Post by cs702 »

gwern wrote: post #32
cs702 wrote: post #29

Earlier quoted context omitted.

The model is sparsely-gated, not sparse. The individual experts in each mixture of experts are dense layers but they're sparsely activated, i.e., on each forward pass only some of them are conditionally used. As to comparing parameter counts, I disagree with you. I think it's perfectly OK to compare parameter counts for different kinds of models. It would also be perfectly OK to compare, say, computational efficiency…

Probably the most relevant comparison here would be a mix of wallclock-hours and FLOPS. The MoE may be inefficient on a parameter level, but it may be the most efficient way to convert FLOPS into model power (sort of like how you currently do better making models wider than deeper - experts are the ultimate 'width').

It depends on your goal. If you want to measure the number of "artificial synapses" or connections, total parameters is the right figure to use, because each weight is one such connection. If you want to measure the computational cost of training or inference, then wallclock-hours and FLOPs would be better figures.

The 100's of trillions of connections (synapses) in the human brain are sparsely used -- i.e., your entire brain doesn't light up in response to every single stimulus. But we still talk about 100's of trillions of synapses when we refer to the size of the human brain's connectome. It's a perfectly valid way of measuring model size.

More to your point, the authors measure the computational cost of training in Table 3 of the paper in TPU-core-years for the various mixture-of-expert models, and compare them to an always-densely-used variant.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#34

Post by ma2rten »

cs702 wrote: post #29
ma2rten wrote: post #25

Earlier quoted context omitted.

This is a sparse model. You can't just compare the parameter count against dense models like GPT-3. Otherwise, Google already had a 137B parameter model in 2017: https://arxiv.org/abs/1701.06538

The model is sparsely-gated, not sparse. The individual experts in each mixture of experts are dense layers but they're sparsely activated, i.e., on each forward pass only some of them are conditionally used. As to comparing parameter counts, I disagree with you. I think it's perfectly OK to compare parameter counts for different kinds of models. It would also be perfectly OK to compare, say, computational efficiency…

Fair enough, sparse usually means weights are sparse and not activations.

Obviously you can compare parameter count if you really want to, but from a technical point of view training a densely activated model is a much bigger feat. Also, I have personally spoken to one of the authors of this paper and they said sparsely activated models tend to well better on tasks that require knowledge but not tasks that require intelligence (e.g. GLUE).

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#35

Post by cs702 »

ma2rten wrote: post #34
cs702 wrote: post #29

Earlier quoted context omitted.

The model is sparsely-gated, not sparse. The individual experts in each mixture of experts are dense layers but they're sparsely activated, i.e., on each forward pass only some of them are conditionally used. As to comparing parameter counts, I disagree with you. I think it's perfectly OK to compare parameter counts for different kinds of models. It would also be perfectly OK to compare, say, computational efficiency…

Fair enough, sparse usually means weights are sparse and not activations. Obviously you can compare parameter count if you really want to, but from a technical point of view training a densely activated model is a much bigger feat. Also, I have personally spoken to one of the authors of this paper and they said sparsely activated models tend to well better on tasks that require knowledge but not tasks that require in…

I agree, training a dense model with the same number of parameters would be much a bigger feat.

Otherwise, as I mentioned elsewhere on this page, we routinely describe the size of the human brain in terms of numbers of synapses (connections), even though they are sparsely activated. Only a small subset of your brain 'lights up' for a given input. Number of parameters (connections) is a perfectly sensible way to measure model size.

Anyway, I expect we will see both much larger sparsely and densely activated models going forward. We live in interesting times :-)

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#36

Post by Veedrac »

xbmcuser wrote: post #13
cs702 wrote: post #7

"Quién es más macho?" In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target.…

Are they using this for google translate yet. As https://www.deepl.com/en/translator is better than google translate currently. Although for translating forums on a website etc I think netflix method would be better I hope google adopts it for its translate app https://arxiv.org/abs/2005.11197

Google Translate's last major update was in May. They still use more restricted model sizes than the top-end research, but the techniques are making their way into the product.

https://ai.googleblog.com/2020/06/recent-advances-in-google-...

With a cursory analysis, it's not obvious whether DeepL is better than Google Translate any more.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#37

Post by Veedrac »

visarga wrote: post #22
teruakohatu wrote: post #20

The brain has ~100+ trillion synapses [1] (There seems to be estimates from 100-1000 T). A 1 trillion parameter model should not be far off, which is about the same number of synapses as house mice. We will be around 1% of the way to human brain complexity (Well, probably not but it is fun to think of it). [1] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

You can't directly compare biological and artificial neurons like that. Biological ones have synapses that function in a much more complex way than weights in a neural net, but are also much slower and noisy. On the other hand, we don't have a robot body to house the model in. Without embodiment it won't be able to learn to interact with the world like us. Thirdly, in humans, specific priors have been baked in the br…

Biological neurons are complex networks of thousands of synapses, and it's definitely reasonably to say a biological neuron is not 1:1 comparable to an artificial NN neuron. Biological neurons can compute XOR[1] and some even contain loops, called autapses.

However it seems fairly reasonable to say a synapse is roughly 1:1 comparable to a network parameter, in that they seem to be doing about the same sort of weighted propagation with about the same computational power. A synapse does work very differently, and has a couple of very low bandwidth side-channels, but its main job is the same job as a network weight.

[1] https://science.sciencemag.org/content/367/6473/83

Post reply on HN