GShard: Scaling giant models with conditional computation and automatic sharding • Hacker News Board

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#21

Post by teej » Thu, Jul 02, 2020, 3:39 AM UTC

saltking112 wrote: post #16

dig6x wrote: post #11

Earlier quoted context omitted.

Launched in 2014, its basically a purpose-built SQL cloud data warehouse solution. Its success pivoted among other factors, on its ability to abstract compute power and data storage to create a modular solution that could be made efficient for any data warehousing configuration. In 2013 AWS augmented its core cloud offering with the introduction of Redshift, a ‘data warehousing as a service’ solution. The Redshift so…

ic. but from Amazon's perspective, if customers want something that is mostly turn-key with the ability to customize, wouldn't they just combine AWS services themselves? I would believe Amazon has DB only solutions, compute only solutions like EC2 etc... So why was Snowflake able to thrive in this environment? Was the market simply too big?

What snowflake offers would cost you millions in engineering time to recreate from AWS primitives from scratch. AWS does offer a competing all-in-one offering (Redshift) but Snowflake has a superior cost structure and has outpaced it on features.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#22

Post by visarga » Thu, Jul 02, 2020, 4:16 AM UTC

teruakohatu wrote: post #20

The brain has ~100+ trillion synapses [1] (There seems to be estimates from 100-1000 T). A 1 trillion parameter model should not be far off, which is about the same number of synapses as house mice. We will be around 1% of the way to human brain complexity (Well, probably not but it is fun to think of it). [1] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

You can't directly compare biological and artificial neurons like that. Biological ones have synapses that function in a much more complex way than weights in a neural net, but are also much slower and noisy.

On the other hand, we don't have a robot body to house the model in. Without embodiment it won't be able to learn to interact with the world like us.

Thirdly, in humans, specific priors have been baked in the brain by evolution (data symmetries and efficiencies). We don't know all of them yet and how to replicate. We do rely on translation invariance for images and time shift invariance for sequences, and permutation invariance for some set and graph neural nets, but they are not all the priors the brain makes use of.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#23

Post by enchiridion » Thu, Jul 02, 2020, 5:20 AM UTC

visarga wrote: post #22

teruakohatu wrote: post #20

The brain has ~100+ trillion synapses [1] (There seems to be estimates from 100-1000 T). A 1 trillion parameter model should not be far off, which is about the same number of synapses as house mice. We will be around 1% of the way to human brain complexity (Well, probably not but it is fun to think of it). [1] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

You can't directly compare biological and artificial neurons like that. Biological ones have synapses that function in a much more complex way than weights in a neural net, but are also much slower and noisy. On the other hand, we don't have a robot body to house the model in. Without embodiment it won't be able to learn to interact with the world like us. Thirdly, in humans, specific priors have been baked in the br…

Do we really rely on translation invariance significantly? It seems more like we scan a very focused area quickly

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#24

Post by modeless » Thu, Jul 02, 2020, 5:57 AM UTC

The most important advancements in machine learning for the next 10 years at least will be in hardware, and the software to take advantage of said hardware. You could even say that was already true starting with AlexNet, but it's even more obvious now with these enormous models.

We've barely scratched the surface of what's possible. Even if Moore's Law was dead (though it seems that TSMC may keep it alive for a bit longer) there are huge gains to be had when co-designing models and hardware. Stuff like https://www.cerebras.net/ is the direction I expect things to go.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#25

Post by ma2rten » Thu, Jul 02, 2020, 7:03 AM UTC

cs702 wrote: post #7

"Quién es más macho?" In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target.…

This is a sparse model. You can't just compare the parameter count against dense models like GPT-3.

Otherwise, Google already had a 137B parameter model in 2017: https://arxiv.org/abs/1701.06538

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#26

Post by tehsauce » Thu, Jul 02, 2020, 8:20 AM UTC

rahimnathwani wrote: post #12

MrUssek wrote: post #5

Earlier quoted context omitted.

There's an extremely large number of parameters per "neuron". The 600B parameters will take up more than 1TB of space in memory, far too much for the 2080 TI or even main memory for most systems.

I'm not talking about inference on a 600B parameter model. GP said they can't do inference on a 32-layer, 2048 neurons-per-layer network. Let's assume every layer is fully connected. So each neuron will have 2048 parameters. So that's 32 * 2048 * 2048 parameters. That's 132MM parameters in 11GB of RAM, or 82 bytes per parameter. If each parameter is 4 bytes (that seems like a lot of precision), plus 4 bytes per calcu…

2048 neurons per layer isn't really an accurate description, what he means is 2048 dimensional embeddings at each layer. The actual multihead attention layers in a transformer are not just feed forward 2048*2048, but actually have many more parameters. That's why there's 600B total.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#27

Post by ddorian43 » Thu, Jul 02, 2020, 10:14 AM UTC

dig6x wrote: post #11

saltking112 wrote: post #9

Earlier quoted context omitted.

sounds interesting. can you elaborate? Not familiar with what Snowflake does or how it compares. Thanks

Launched in 2014, its basically a purpose-built SQL cloud data warehouse solution. Its success pivoted among other factors, on its ability to abstract compute power and data storage to create a modular solution that could be made efficient for any data warehousing configuration. In 2013 AWS augmented its core cloud offering with the introduction of Redshift, a ‘data warehousing as a service’ solution. The Redshift so…

What is the cost when getting all data from network compared to local disks ? Is this connected to the cloud not offering local persistent disks ? Will this still work in colocation ?

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#28

Post by dna_polymerase » Thu, Jul 02, 2020, 10:40 AM UTC

modeless wrote: post #24

The most important advancements in machine learning for the next 10 years at least will be in hardware, and the software to take advantage of said hardware. You could even say that was already true starting with AlexNet, but it's even more obvious now with these enormous models. We've barely scratched the surface of what's possible. Even if Moore's Law was dead (though it seems that TSMC may keep it alive for a bit l…

Hardware will be a huge part, yes, but algorithmic advances would be even better. Utilizing existing commodity hardware to full extent is where the money is at. Specialized hardware will probably remain just that, specialized and mostly too expensive.

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#29

Post by cs702 » Thu, Jul 02, 2020, 1:53 PM UTC

ma2rten wrote: post #25

cs702 wrote: post #7

"Quién es más macho?" In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target.…

This is a sparse model. You can't just compare the parameter count against dense models like GPT-3. Otherwise, Google already had a 137B parameter model in 2017: https://arxiv.org/abs/1701.06538

The model is sparsely-gated, not sparse. The individual experts in each mixture of experts are dense layers but they're sparsely activated, i.e., on each forward pass only some of them are conditionally used.

As to comparing parameter counts, I disagree with you. I think it's perfectly OK to compare parameter counts for different kinds of models. It would also be perfectly OK to compare, say, computational efficiency per parameter in each forward pass (which for this model is impressive), but that wasn't the focus on my comment above.

Finally, you're right that I didn't mention all the interim parameter counts that we have seen below 600B in all transformer variants. The list would have been way too long had I tried to include every figure!

Re: GShard: Scaling giant models with conditional computation and automatic sharding

#30

Post by cs702 » Thu, Jul 02, 2020, 2:02 PM UTC

IfOnlyYouKnew wrote: post #18

cs702 wrote: post #7

"Quién es más macho?" In a very short time, transformers have gone from under 1B, to 1.5B, to 3B, to 5B, to 175B, and now 600B parameters. 1T is only, what, like 67% more parameters, and therefore likely to be achieved in the short term. In fact, the authors of this paper tried 1T but ran into numerical issues that they will surely address soon. Not long after someone crosses 1T, expect 10T to become the next target.…

Well if you can come up with a more elegant way forward, one that doesn’t require all that hardware and money and should, therefore, by definition be within the reach of the critics of “big” AI, I see no reason why it’s superior qualitative results wouldn’t be appreciated.

I'm not a critic! But I can see how my comment, meant to be a bit humorous, could be misinterpreted as being critical.

Personally, I think the friendly race to build bigger models is a great development. As I mentioned above, it seems to be leading to models that generate text/sequences that are qualitatively much better.