GShard: Scaling giant models with conditional computation and automatic sharding
1–10 of 37 posts
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#2Yet another paper with results that basically look like this:
https://d3b8hk1o42ev08.cloudfront.net/wp-content/uploads/201...
Still impressive, don't get me wrong, but I am starting to believe that NLP will be dominated increasingly by the big players since they are the only ones who can train a 1 TRILLION parameter model (they show that in the paper). I can't even do inference with a 36 layer, 2048 neuron per layer network with my GTX 2080ti. Sad....
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#3Yet another paper with results that basically look like this: https://d3b8hk1o42ev08.cloudfront.net/wp-content/uploads/201... Still impressive, don't get me wrong, but I am starting to believe that NLP will be dominated increasingly by the big players since they are the only ones who can train a 1 TRILLION parameter model (they show that in the paper). I can't even do inference with a 36 layer, 2048 neuron per layer…
Stay tuned for algorithmic advancements.
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#4Yet another paper with results that basically look like this: https://d3b8hk1o42ev08.cloudfront.net/wp-content/uploads/201... Still impressive, don't get me wrong, but I am starting to believe that NLP will be dominated increasingly by the big players since they are the only ones who can train a 1 TRILLION parameter model (they show that in the paper). I can't even do inference with a 36 layer, 2048 neuron per layer…
"I can't even do inference with a 36 layer, 2048 neuron per layer network with my GTX 2080ti."
Not even for a single instance? Your GPU has 11GB of RAM. Why isn't 14k per neuron enough? Is the input really large, or does each neuron have very high precision?
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#5Yet another paper with results that basically look like this: https://d3b8hk1o42ev08.cloudfront.net/wp-content/uploads/201... Still impressive, don't get me wrong, but I am starting to believe that NLP will be dominated increasingly by the big players since they are the only ones who can train a 1 TRILLION parameter model (they show that in the paper). I can't even do inference with a 36 layer, 2048 neuron per layer…
"I can't even do inference with a 36 layer, 2048 neuron per layer network with my GTX 2080ti." Not even for a single instance? Your GPU has 11GB of RAM. Why isn't 14k per neuron enough? Is the input really large, or does each neuron have very high precision?
There's an extremely large number of parameters per "neuron". The 600B parameters will take up more than 1TB of space in memory, far too much for the 2080 TI or even main memory for most systems.
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#6"...600 billion parameters using automatic sharding. We demonstrate that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art."
It does appear that at the initial, resource intensive stages of tech like NLP big tech is primed to pave the way. We saw this happen across cloud, AI more generally, storage etc. But big tech then begins focusing on making the tech accessible to industry value chains (Azure, AWS, Amazon's AI services etc.). But as the industry matures there's more room for specialized startups/companies to enter the space to capture lucrative niches - thats exactly what Snowflake did for Cloud.
Definitely see this kind of scale as a step toward a more robust, mature industry if anything. Better it move forward than not.
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#7"Quién es más macho?"
In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target. And why not? The best-funded AI research groups are in a friendly competition to build the biggest, baddest, meanest m-f-ing models the world has ever seen.
Scores continue to increase with diminishing returns, which is all fine and nice, but more importantly it seems we should expect to see machine-generated text getting much better from a qualitative standpoint -- that is, becoming less and less distinguishable from a lot of human output. That has been the trend so far.
We live in interesting times.
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#8Awe inspiring thinking of the number of transistors working in orchestra to translate human language to english...
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#9"...600 billion parameters using automatic sharding. We demonstrate that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art." It does appear that at the initial, resource intensive stages of tech like NLP big tech is primed to pave the way. We saw this happen across cloud, AI more g…
sounds interesting. can you elaborate? Not familiar with what Snowflake does or how it compares. Thanks
Re: GShard: Scaling giant models with conditional computation and automatic sharding
#10"...600 billion parameters using automatic sharding. We demonstrate that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art." It does appear that at the initial, resource intensive stages of tech like NLP big tech is primed to pave the way. We saw this happen across cloud, AI more g…
sounds interesting. can you elaborate? Not familiar with what Snowflake does or how it compares. Thanks
Similarly, not familiar with the unit economics when it's sitting on top of S3 / Azure storage. But it's growing really quick so it must be Something.