Really cool!
I'd love to see a table comparing the results against the other gigantic models (I know could Google the other results and merge them together but no thanks)
Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
11–20 of 100 posts
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#12So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…
Unless we have misunderstood neurons, and microtubules are the fundamental computational unit in which case we are out by an order of magnitude
There was a result recently of modeling an organic neuron with 1000 digital neurons.
And even if that result was perfect modeling of the neuron, that assumes perfect and exhaustive data readings on the organic neuron, which is, frankly, unlikely. (Not that I know how to estimate how much it's missing, but I don't think we fully understand a single neuron yet.)
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#13Interesting that books3 and The Pile are among the largest corpus used for training - both with copyright concerns.
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#14So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…
A 10 trillion parameter model was mentioned here: https://mobile.twitter.com/ethancaballero/status/14458268620...
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#15So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…
A 10 trillion parameter model was mentioned here: https://mobile.twitter.com/ethancaballero/status/14458268620...
That's MoE.
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#16This reminds me a little bit of the early 2000's where search engines would list the number of indexed pages on their homepage. For language models, does large = good? I'm guessing the quality of the corpus matters as much.
Maybe? The Scaling Hypothesis[1] suggests that greater capabilities of intelligence may emerge from scaling up 'scalable architectures' to large sizes. GPT-3 exhibits 'meta-learning' capabilities that GPT-2 did not (like learning how to sum numbers)--probably just because its a 100x larger version of GPT-2.
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#17Has there been any update on the legality of using this kind of model? Is it ok to just crawl the web, take any content you want, train a model and sell access to the model like OpenAI/GPT-3/GitHub Copilot?
For the most part, everything that's not barred by law is "legal." Does this use constitute copyright infringement (if it were trained on copyrighted material)? IMO no, but it depends very much on the use of the model. Copilot is especially interesting because instead of being used for simple inference the model is being used to author new works that might aspire to also be copyrighted. Are those new works derivative works? Perhaps. We consider art and science produced by humans to be inspired in part by that which they've been exposed to before. If the model hasn't been overfitted, it should generalize its 'knowledge' sufficiently that it's 'similar' to our intelligence. Humans can commit copyright infringement when they recall and author content so specifically as to be a derived work.
In any case: my opinion matters for naught. The only 'update' you'd get that matters is from a court producing a ruling. Legal journals might chime in but their opinion isn't binding. Theoretically there could be legislation to clarify but that's probably a really, really, really long way off.
Certainly some of the training looks to be content that's not copyrighted or no longer copyrighted, btw.
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#18This is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#19Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
For some reason this issue with model having insane amounts of weights but training data being small is not something that is an issue for modern NNs.
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#20This is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?
When you say "inference", do you mean "interface", or is "inference" an ML term I'm not familiar with?