Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GShard: Scaling giant models with conditional computation and automatic sharding

arxiv.org

Post reply on HN
Print view View original

11–20 of 37 posts

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#11

Post by dig6x »

saltking112 wrote: post #9
dig6x wrote: post #6

"...600 billion parameters using automatic sharding. We demonstrate that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art." It does appear that at the initial, resource intensive stages of tech like NLP big tech is primed to pave the way. We saw this happen across cloud, AI more g…

sounds interesting. can you elaborate? Not familiar with what Snowflake does or how it compares. Thanks

Launched in 2014, its basically a purpose-built SQL cloud data warehouse solution. Its success pivoted among other factors, on its ability to abstract compute power and data storage to create a modular solution that could be made efficient for any data warehousing configuration.

In 2013 AWS augmented its core cloud offering with the introduction of Redshift, a ‘data warehousing as a service’ solution. The Redshift solution bundled compute and storage, reducing the ability to meet individual customer needs to scale either component separately in a cost efficient manner. Not having the option to unbundle compute and storage was inconsistent with the flexible nature that cloud had become known for.

Snowflake’s solution separated storage, compute, and services into separate layers, allowing them to scale independently and achieve greater cost efficiencies. By offering flexibility it was able to better address the requirements of a wider range of customers - who had previously been limited to the more restrictive bundled options, like Redshift.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#12

Post by rahimnathwani »

MrUssek wrote: post #5
rahimnathwani wrote: post #4

Earlier quoted context omitted.

"I can't even do inference with a 36 layer, 2048 neuron per layer network with my GTX 2080ti." Not even for a single instance? Your GPU has 11GB of RAM. Why isn't 14k per neuron enough? Is the input really large, or does each neuron have very high precision?

There's an extremely large number of parameters per "neuron". The 600B parameters will take up more than 1TB of space in memory, far too much for the 2080 TI or even main memory for most systems.

I'm not talking about inference on a 600B parameter model. GP said they can't do inference on a 32-layer, 2048 neurons-per-layer network. Let's assume every layer is fully connected. So each neuron will have 2048 parameters. So that's 32 * 2048 * 2048 parameters. That's 132MM parameters in 11GB of RAM, or 82 bytes per parameter. If each parameter is 4 bytes (that seems like a lot of precision), plus 4 bytes per calculated value, you're still only using 10% of the GPU's RAM. You should be able to do inference on a batch of 16-20 examples at a time.

What have I missed?

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#13

Post by xbmcuser »

cs702 wrote: post #7

"Quién es más macho?" In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target.…

Are they using this for google translate yet. As https://www.deepl.com/en/translator is better than google translate currently. Although for translating forums on a website etc I think netflix method would be better I hope google adopts it for its translate app https://arxiv.org/abs/2005.11197

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#14

Post by cs702 »

xbmcuser wrote: post #13
cs702 wrote: post #7

"Quién es más macho?" In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target.…

Are they using this for google translate yet. As https://www.deepl.com/en/translator is better than google translate currently. Although for translating forums on a website etc I think netflix method would be better I hope google adopts it for its translate app https://arxiv.org/abs/2005.11197

Highly unlikely at the moment. But clearly that is the direction in which translation is going, so companies lacking the economies of scale that come with owning massive computational infrastructure will be at a serious disadvantage.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#16

Post by saltking112 »

dig6x wrote: post #11
saltking112 wrote: post #9

Earlier quoted context omitted.

sounds interesting. can you elaborate? Not familiar with what Snowflake does or how it compares. Thanks

Launched in 2014, its basically a purpose-built SQL cloud data warehouse solution. Its success pivoted among other factors, on its ability to abstract compute power and data storage to create a modular solution that could be made efficient for any data warehousing configuration. In 2013 AWS augmented its core cloud offering with the introduction of Redshift, a ‘data warehousing as a service’ solution. The Redshift so…

ic. but from Amazon's perspective, if customers want something that is mostly turn-key with the ability to customize, wouldn't they just combine AWS services themselves? I would believe Amazon has DB only solutions, compute only solutions like EC2 etc... So why was Snowflake able to thrive in this environment? Was the market simply too big?

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#17

Post by dig6x »

saltking112 wrote: post #16
dig6x wrote: post #11

Earlier quoted context omitted.

Launched in 2014, its basically a purpose-built SQL cloud data warehouse solution. Its success pivoted among other factors, on its ability to abstract compute power and data storage to create a modular solution that could be made efficient for any data warehousing configuration. In 2013 AWS augmented its core cloud offering with the introduction of Redshift, a ‘data warehousing as a service’ solution. The Redshift so…

ic. but from Amazon's perspective, if customers want something that is mostly turn-key with the ability to customize, wouldn't they just combine AWS services themselves? I would believe Amazon has DB only solutions, compute only solutions like EC2 etc... So why was Snowflake able to thrive in this environment? Was the market simply too big?

Yeh the CLoud market was at a stage where the niche with some convenience add could thrive. Now we're seeing all these multi cloud platforms emerge because enterprises are managing multiple server providers at once etc. so you can imagine all the opportunities for horizontal scaling beyond big tech in the industry.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#18

Post by IfOnlyYouKnew »

cs702 wrote: post #7

"Quién es más macho?" In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target.…

Well if you can come up with a more elegant way forward, one that doesn’t require all that hardware and money and should, therefore, by definition be within the reach of the critics of “big” AI, I see no reason why it’s superior qualitative results wouldn’t be appreciated.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#19

Post by [deleted] »

rahimnathwani wrote: post #12
MrUssek wrote: post #5

Earlier quoted context omitted.

There's an extremely large number of parameters per "neuron". The 600B parameters will take up more than 1TB of space in memory, far too much for the 2080 TI or even main memory for most systems.

I'm not talking about inference on a 600B parameter model. GP said they can't do inference on a 32-layer, 2048 neurons-per-layer network. Let's assume every layer is fully connected. So each neuron will have 2048 parameters. So that's 32 * 2048 * 2048 parameters. That's 132MM parameters in 11GB of RAM, or 82 bytes per parameter. If each parameter is 4 bytes (that seems like a lot of precision), plus 4 bytes per calcu…

[deleted]

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#20

Post by teruakohatu »

The brain has ~100+ trillion synapses [1] (There seems to be estimates from 100-1000 T).

A 1 trillion parameter model should not be far off, which is about the same number of synapses as house mice.

We will be around 1% of the way to human brain complexity (Well, probably not but it is fun to think of it).

[1] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

Post reply on HN