11–20 of 87 posts
What does GPT mean? I assume it is not about partition tables (GUID Parition Table), it has something to do with NLP, but besides that it is hard to find what does this acronym mean.
What does GPT mean? I assume it is not about partition tables (GUID Parition Table), it has something to do with NLP, but besides that it is hard to find what does this acronym mean.
All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…
I mostly agree with you. However, > It's a significant experimental accomplishment that points to a future in which general-purpose NLP models could be used for novel tasks without requiring additional training from the get-go. This premise is still purely science fiction. This model does not touch on either novel tasks nor being free from pretraining (unless I misunderstand). But overall, I think you’re right: it’s…
All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…
Moore's law is running on fumes at this point. The complexity of further scaling has reached geopolitical proportions. We need to get back to looking at more creative models in both the software and hardware domains.
the model achieves competitive results on many NLP tasks and benchmarks without finetuning
The article dismisses this result with the following analogy:
“No, my 10-year-old math prodigy hasn’t proven any new theorems, but she can get a perfect score on the math SAT in under 10 minutes. Isn’t that groundbreaking?”
And I tend to agree. We've had a game of benchmarking brinkmanship for a while now. At what point are we going to see some groundbreaking applications?
Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2
All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…
I'm reminded of Rich Sutton's essay, "The Bitter Lesson:" Moore's law is running on fumes at this point. The complexity of further scaling has reached geopolitical proportions. We need to get back to looking at more creative models in both the software and hardware domains. the model achieves competitive results on many NLP tasks and benchmarks without finetuning The article dismisses this result with the following a…
Also, Moore's Law is running on fumes, yes, but there's quite a bit of R&D focused on coming up with hardware that massively scales up (e.g., by more efficiently parallelizing) the dense and sparse multiply-sum operations common to so many AI models. I think Sutton's point about models that leverage computation is spot-on.
> Transformers are extremely interesting. And this is about the least interesting transformer paper one can imagine in 2020. Because it's not a transformer paper. This paper goal was to see how far can an increase in compute continue to deliver an increase in model performance. There is no better way to study this than to take a very well known architecture and keep it the same as possible, otherwise it becomes very…
All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…
This paper implements an architecture that will be out of reach for me for about 5 years. So, I ask myself "why will this paper matter in 5 years?"
There are two reasons I can imagine:
1. It shows that there is no phase change in the size-performance trend already documented over many orders of magnitude.
2. It was used as input data for pruning or distillation algorithms rooted in a better understanding of why language models work.
If the NLP community remains laser focused on hill-climbing compute-agnostic benchmarks, I don't think there will be enough people working on 2.
If 1 is the only reason it matters, I struggle to see how it is worth the cost in high-end research talent. It feels like standard low-risk corporate iteration.
All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…
Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2