GPT-3: A Disappointing Paper? • Hacker News Board

Re: GPT-3: A Disappointing Paper?

#41

Post by jeeceebees » Sat, May 30, 2020, 5:49 PM UTC

rmrfstar wrote: post #18

cs702 wrote: post #7

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…

> practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering This paper implements an architecture that will be out of reach for me for about 5 years. So, I ask myself "why will this paper matter in 5 years?" There are two reasons I can imagine: 1. It shows that there is no phase change in the size-performance trend already documented over many orders of magnitude. 2.…

I think the larger models get, the more incentive there is for researchers to look into pruning/distilling them for practical use.

GPT-1,2,3 et al. have all shown that larger is better. While in the short term this means people will simply throw larger and larger clusters at the problem, in the longer term there needs to be inovation in making it more efficient on the clusters we have (as even the cloud has limits).

I think sheer parameter count is an important part of the equation in general intelligence, so it's important that there are labs that work on scaling up promising leads to trillions of parameters on top of labs thinking of new promising directions.

Re: GPT-3: A Disappointing Paper?

#42

Post by The_rationalist » Sat, May 30, 2020, 6:07 PM UTC

psb217 wrote: post #20

The_rationalist wrote: post #2

Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2

FYI, the MELBO bound in that paper is invalid. Their perplexity numbers using the MELBO bound are also invalid.

Where is the error? How much would that change their score of 4.6?

Re: GPT-3: A Disappointing Paper?

#43

Post by [deleted] » Sat, May 30, 2020, 6:13 PM UTC

cs702 wrote: post #40

rmrfstar wrote: post #18

Earlier quoted context omitted.

> practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering This paper implements an architecture that will be out of reach for me for about 5 years. So, I ask myself "why will this paper matter in 5 years?" There are two reasons I can imagine: 1. It shows that there is no phase change in the size-performance trend already documented over many orders of magnitude. 2.…

> "Why will this paper matter in 5 years?" 3. First evidence that performance continues to improve at hundreds of billions of weights -- paving the way for trillions of weights, approaching orders of magnitude comparable to that of the human brain connectome. 4. First evidence (AFAIK) that larger NLP models do not need task-specific finetuning -- paving the way for general-purpose models that work well on any NLP tas…

[deleted]

Re: GPT-3: A Disappointing Paper?

#44

Post by gambler » Sat, May 30, 2020, 6:18 PM UTC

Article> One of their experiments, “Learning and Using Novel Words,“ strikes me as more remarkable than most of the others and the paper’s lack of focus on it confuses me.

This sort of "learning" is not necessarily real learning and it's not new for GPT-3. Even reduced GPT-2 willingly used made-up terms from the prompt in its results:

https://medium.com/@VictorBanev/interrogating-gpt-2-345m-aaf...

Search the article for 'Now I will feed it the same thing, but with a bunch of made-up terms.' It has some examples of how that stuff worked.

I've already posted this in the original discussion of GPT-3 paper and I will post it again: statements about whether some system "learns new words" or "does math" require hypothesis formulation and testing. It astounds me that many people in ML community not only don't do these sort of things, but even actively oppose to the very idea of them being necessary.

Recently there was a great live-stream from DarkHorse talking about this problem in science in general:

https://www.youtube.com/watch?v=QvljruLDhxY

They talk about "data-driven" science and the fundamental problems with that notion.

Re: GPT-3: A Disappointing Paper?

#45

Post by KKKKkkkk1 » Sat, May 30, 2020, 6:30 PM UTC

cs702 wrote: post #7

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft.

Rich Sutton is a great scientist but he is fooled by randomness. He initiated his research program just as Moore's law was taking off. Thanks to Moore, his approach saw incredible success and brought him deserved acclaim. But just as Moore's law is pulling the rug from under him, he is using his stature to claim that no other approach but his can work.

Re: GPT-3: A Disappointing Paper?

#46

Post by justicezyx » Sat, May 30, 2020, 6:40 PM UTC

KKKKkkkk1 wrote: post #45

cs702 wrote: post #7

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his r…

The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.

Re: GPT-3: A Disappointing Paper?

#47

Post by gentleman11 » Sat, May 30, 2020, 6:48 PM UTC

cs702 wrote: post #7

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…

Think about the waves of fake news to come. Soon every corporation (and political group) in American will have a data center dedicated to overloading social media with messaging that benefits them. At least right now it’s not so hard to spot the bots.

I am having some trouble seeing the benefits in lieu of the mess that is coming.

Re: GPT-3: A Disappointing Paper?

#48

Post by papeda » Sat, May 30, 2020, 6:49 PM UTC

cs702 wrote: post #7

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…

> More generally, I find that some AI researchers and practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering. I think this tendency is misguided. We must build giant machines and gather experimental evidence from them -- akin to physicists who build giant high-energy particle colliders to gather experimental evidence from them.

The analogy is a bit off to me. As far as I can tell, there was significant impetus from within particle physics to commit a huge amount of resources and political effort toward verifying theories with experiment. I don't see anything similar in deep learning, because in this case the "theory" is mostly that "bigger is probably better". I think that idea is pretty uncontroversial for stuff like this. And if the work reduces to marshaling enough resources, what exactly is it?

We should give OpenAI some credit for doing the damn thing, but as is the result kind of seems like an answer to a question that people weren't really asking.

Re: GPT-3: A Disappointing Paper?

#49

Post by [deleted] » Sat, May 30, 2020, 7:22 PM UTC

reddickulous wrote: post #24

It would be cool if there was a platform to crowd source compute resources to train stuff like this so that regular people (without 7 figure budgets) can have access to these models which are becoming increasingly out of reach to the general public.

[deleted]

Re: GPT-3: A Disappointing Paper?

#50

Post by tal8d » Sat, May 30, 2020, 7:25 PM UTC

chongli wrote: post #29

tal8d wrote: post #22

Earlier quoted context omitted.

> We need to get back to looking at more creative models in both the software and hardware domains. That would be pretty foolish, given the fact that every hand crafted model eventually gets surpassed with brute force. A better use of time would be tackling whatever you mean by "complexity of further scaling has reached geopolitical proportions". I'm not a fan of it, as it is terribly inelegant, but denying the years…

That would be pretty foolish, given the fact that every hand crafted model Who said anything about hand crafted AI models? I’m talking about revisiting our models of computation. Moore’s law has long made it impossible to challenge the dominance of Von Neumann. Perhaps what we need to make further progress is some sort of decentralized, busless computer? Who knows?

If that was the first time you'd seen mention of that essay, then you'd be forgiven for not knowing that always precedes discussion of hand crafting vs brute force. I've never seen it be answered with a call for exotic computation, so my mistake. I've seen plenty of energy requirement estimates related to defeating various cryptographic algorithms, using spherical cows, etc. You'll need a lot more than an architectural change to make a noticeable dent, you'll need the discovery and industrialization of new physics - not the sort of thing you want to hang your hopes on.