China's WuDao model had 1.75T parameters and Google's Switch transformer had 1.6T. How is this the world's largest then?
Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
61–70 of 100 posts
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#62Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#63Will anyone outside of Nvidia be able to access it? GPT-3 at least has an API.
(Team member of this project)
Just a clarification, both Microsoft and Nvidia have ownership of this model. Here is the Microsoft version of same announcement.
https://www.microsoft.com/en-us/research/blog/using-deepspee...
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#64all these models are over-hyped. we are nowhere close to AGI until we can come up with a reasonable definition for consciousness
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#65Any idea if they’ll release an API similar to GPT-3? It’s great that larger and larger models are trained but without enabling access to the trained models developers are left out from the progress…
I hope they don’t release an API the way they released an API for GPT-3.
Why? What do you do not like about GPT-3 API?
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#66How has the previous largest model, gpt3, generateda value? How much better is this model at those tasks?
GPT-3 powers GitHub Copilot, so it generated some value.
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#67Earlier quoted context omitted.
I hope they don’t release an API the way they released an API for GPT-3.
Why? What do you do not like about GPT-3 API?
Can't fine tune it. Can't use it on private data. Pay per symbol. High latency access. Inconsistent performance. No internet disconnected use. Can't guarentee repeatable results. Can't (easily) replace the sampling with alternatives that have different behavior (e.g. for using it for compression).
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#68What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?
https://arxiv.org/pdf/1802.10217.pdf
This is the paper I love to link in response to these sort of objections.
Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
#69Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
A single weight can’t encode an individual word, but the ratio looks close to overfitting too me too.