GPT-3: A Disappointing Paper?
greaterwrong.com
GPT-3: A Disappointing Paper?
1–10 of 87 posts
Re: GPT-3: A Disappointing Paper?
#2Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank
(4.6 vs 20) and with order of magnitudes less parameters
is talked about nowhere on the web...
Re: GPT-3: A Disappointing Paper?
#3Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2
I've seen you post this before and you seem enthusiastic about it. Do you have a good summary of why you feel this paper so important?
Re: GPT-3: A Disappointing Paper?
#4> “GPT-3″ is just a bigger GPT-2. In other words, it’s a straightforward generalization of the “just make the transformers bigger” approach
Yes it’s true. But there is a difference between what’s interesting and what works. deep learning (RNNs, transformers, etc.) is usually old ideas applied at large scale with slight modifications. Proving a model works well at large scale (175B parameters) is a great contribution and measures our progress towards AI.
Re: GPT-3: A Disappointing Paper?
#5Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2
Any models or code available?
Re: GPT-3: A Disappointing Paper?
#6Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2
I've seen you post this before and you seem enthusiastic about it. Do you have a good summary of why you feel this paper so important?
why you feel this paper so important? the best summary are the benchmarks:
first place at question answering (on yahoo task)
First place on language modeling for pen treebank (by FAR)
So it is a totally new model that will probably keep evolving and being applied to more and more kind of NLP tasks. And it seems that it can have the first place on most NLP tasks, its empirically the breakthrough of the year.
It achieve this while having an extremely small number of parameters which shows: The model is smarter The model has room for more parameter hence even more accuracy!
Finally, theoretically it is a breakthrough as it is a port of a computer vision technology (variational autoencoders) to the NLP world. Actually it might be the successor to the Transformer paradigm. I wonder what pile of incremental improvement will researchers will be able to bring to it like they have on the Transformer paradigm (spanBert, XLnet, etc)
Re: GPT-3: A Disappointing Paper?
#7All valid points, but I disagree with the conclusion, for several reasons:
* First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment.
* Second, the model achieves competitive results on many NLP tasks and benchmarks without finetuning, using only a context window of text for instructions and input. There is only unsupervised (i.e., autoregressive) pretraining. AFAIK, this is the first paper that has reported a model doing this. It's a significant experimental accomplishment that points to a future in which general-purpose NLP models could be used for novel tasks without requiring additional training from the get-go.
* Finally, the model’s text generation fools human beings without having to cherry-pick examples. AFAIK, this is the first paper that has reported a model doing this. It's another significant experimental accomplishment.
More generally, I find that some AI researchers and practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering. I think this tendency is misguided. We must build giant machines and gather experimental evidence from them -- akin to physicists who build giant high-energy particle colliders to gather experimental evidence from them.
I'm reminded of Rich Sutton's essay, "The Bitter Lesson:"
Re: GPT-3: A Disappointing Paper?
#8> Transformers are extremely interesting. And this is about the least interesting transformer paper one can imagine in 2020.
Because it's not a transformer paper.
This paper goal was to see how far can an increase in compute continue to deliver an increase in model performance. There is no better way to study this than to take a very well known architecture and keep it the same as possible, otherwise it becomes very hard to know what is due to the increase size of the model and what is due to the tweaks you make.
So yes, it's a disappointing paper if you expect it to be on a different topic than what it is.
Re: GPT-3: A Disappointing Paper?
#9All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…
I mostly agree with you. However,
> It's a significant experimental accomplishment that points to a future in which general-purpose NLP models could be used for novel tasks without requiring additional training from the get-go.
This premise is still purely science fiction. This model does not touch on either novel tasks nor being free from pretraining (unless I misunderstand). But overall, I think you’re right: it’s significant for a number of reasons.
Re: GPT-3: A Disappointing Paper?
#10What does GPT mean?
I assume it is not about partition tables (GUID Parition Table), it has something to do with NLP, but besides that it is hard to find what does this acronym mean.