Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI training method exceeds GPT-3 performance with fewer parameters

infoq.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: AI training method exceeds GPT-3 performance with fewer parameters

#61

Post by leereeves »

Cybiote wrote: post #54
leereeves wrote: post #41

Earlier quoted context omitted.

Powerful analogy, but analogies are dangerous. They can obscure what's really happening. In this case, by analogy, Ferrari made the comparison to the super cheap SUV from 2003. That is, OpenAI compared GPT3 to BERT on the SuperGLUE benchmark, in the paper announcing GPT3 [1]. They did so to demonstrate GPT3's ability to learn a new task given only a few examples of the task ("few-shot learning"). The limited amount o…

Unfortunately, your correction is still misleading because it fails to capture what really sets GPT-3 apart. Ironically, GPT-3's limitation also highlights its strength. As it is not capable of learning (few shot or otherwise) in the strict sense of permanently changing its parameters based on examples, all its demonstrated capabilities are completely at inference time. It somehow configures itself at inference time…

Number of parameters remains an important practical concern, and an important goal for research (see [1] for a recent example).

It is true that this model requires a large amount of unlabeled data, in addition to a small amount of labeled data, but gathering unlabeled data is often easy.

So I think it's potentially misleading to say "this approach remains at a severe disadvantage when amount of effort and time required to gather data and train something useful is accounted for".

I'd say this approach actually has a serious advantage compared to GPT3, which is locked inside the walls of OpenAI, can only be used with their permission, and is too big for most people to use (let alone train) anyway. The cost and effort to use this approach on large real world problems is probably less than using GPT3.

And it may also have an advantage in terms of the total amount of data and training required, when all of the data and training in the original pretrained models is included. That is, it may be a more efficient way to train new models from scratch for tasks for which few labeled examples are available.

1: https://ai.googleblog.com/2020/09/advancing-nlp-with-efficie...

Re: AI training method exceeds GPT-3 performance with fewer parameters

#62

Post by msamwald »

sailingparrot wrote: post #56
newen wrote: post #51

Earlier quoted context omitted.

No one except people doing work on Cloze questions will be interested in this with that title. Badly written PR puff piece is bad but no one will read that abstract unless they are in that field.

> but no one will read that abstract unless they are in that field Which is fine? What's the point of getting people outside of the field reading a paper by essentially lying about the content? They come expecting A, and if they read it they understand it's actually about B. Loss of time for everyone except the author who wants to make a little buzz.

I think few-shot learning (or priming) is actually the main selling point of GPT-3 for most practical applications (rather than merely entertaining language generation). So if there is a method that achieves the same goal with a model that is simple enough to be used by normal developers and researchers without OpenAI-scale infrastructure, that does seem buzz-worthy.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#63

Post by causality0 »

newen wrote: post #51
cs702 wrote: post #49

The headline is clickbait (GPT-3 is about language modeling/generation, and there's ZERO mention of that in this work), but the actual work is interesting and the paper is worth a read. To the OP: please consider changing the headline to "Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference" and link to the original paper at https://arxiv.org/abs/2001.07676 instead of this PR puf…

No one except people doing work on Cloze questions will be interested in this with that title. Badly written PR puff piece is bad but no one will read that abstract unless they are in that field.

Exactly. "We made GPT-3 with 99.9% less overhead" is a titillating and pants-wetting event. This is not.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#64

Post by whimsicalism »

msamwald wrote: post #62
sailingparrot wrote: post #56

Earlier quoted context omitted.

> but no one will read that abstract unless they are in that field Which is fine? What's the point of getting people outside of the field reading a paper by essentially lying about the content? They come expecting A, and if they read it they understand it's actually about B. Loss of time for everyone except the author who wants to make a little buzz.

I think few-shot learning (or priming) is actually the main selling point of GPT-3 for most practical applications (rather than merely entertaining language generation). So if there is a method that achieves the same goal with a model that is simple enough to be used by normal developers and researchers without OpenAI-scale infrastructure, that does seem buzz-worthy.

> So if there is a method that achieves the same goal with a model that is simple enough to be used by normal developers and researchers without OpenAI-scale infrastructure, that does seem buzz-worthy.

That's trivial and has already been done. GPT-3 didn't even get SOTA on SuperGlue.

These are the sort of misunderstandings that could have been avoided if the title was better.

In general, paper with "new variation of cloze pre-training task for this specific task" is a new section of the literature that is rapidly becoming sort of mundane and uninteresting because there are so many papers doing small variations of the same basic idea.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#65

Post by whimsicalism »

Cybiote wrote: post #54
leereeves wrote: post #41

Earlier quoted context omitted.

Powerful analogy, but analogies are dangerous. They can obscure what's really happening. In this case, by analogy, Ferrari made the comparison to the super cheap SUV from 2003. That is, OpenAI compared GPT3 to BERT on the SuperGLUE benchmark, in the paper announcing GPT3 [1]. They did so to demonstrate GPT3's ability to learn a new task given only a few examples of the task ("few-shot learning"). The limited amount o…

Unfortunately, your correction is still misleading because it fails to capture what really sets GPT-3 apart. Ironically, GPT-3's limitation also highlights its strength. As it is not capable of learning (few shot or otherwise) in the strict sense of permanently changing its parameters based on examples, all its demonstrated capabilities are completely at inference time. It somehow configures itself at inference time…

> It somehow configures itself at inference time so that state machines which produce plausible continuations of whatever pattern it was fed, are most probably generated.

What is this sentence supposed to convey? I'm an NLP practicioner/researcher and this isn't even true - as GPT isn't a "state machine" as the latent space is continuous and not finite.

Moreover, there is nothing that makes GPT-3 "not capable of learning." It has had very exciting results from language modeling a zero-shot task at inference time, but there's nothing (besides compute) precluding fine-tuning of it in principle.

I agree with the rest of your comment.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#66

Post by msamwald »

Veedrac wrote: post #50
lzzzfelipe wrote: post #5

GPT3's strength is on language generation, so using *GLUE for evaluating it (where encoder type models are just better) and claiming to have 99.9% less parameters is sensationalism.

I had this discussion with Timo Schick, the first author, on Reddit. My final comment is copied below, with the relevant context. Note that GPT-3's approach to *GLUE involved no training on the task, just a good choice of prompt, whereas PET and iPET also use fine-tuning. Also, because distillation takes large amounts of training data, they use ensembles in the true few-shot regime, so their parameter efficiency is s…

A shorter reply would be: It would be great to compare PET not only to GPT-3, but also to other models, especially ones geared towards few-shot learning.

Do you know of any other models that should be used for such a comparison, or are there already any relevant results on SuperGLUE that should be mentioned?

Re: AI training method exceeds GPT-3 performance with fewer parameters

#67

Post by frankmcsherry »

fnord123 wrote: post #58
albertwang wrote: post #52

Earlier quoted context omitted.

Out of (self-interested) curiosity, what are you referring to with the "nvme->timely dataflow on a laptop" reference? The closest benchmark I could find was the "FASTER State Management for Timely Dataflow" paper from ETH Zurich, but that wasn't run on a laptop.

http://www.frankmcsherry.org/assets/COST.pdf

It's a bit different, right, because that paper presents better algorithms on the same datasets as the work it cites.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#68

Post by msamwald »

whimsicalism wrote: post #64
msamwald wrote: post #62

Earlier quoted context omitted.

I think few-shot learning (or priming) is actually the main selling point of GPT-3 for most practical applications (rather than merely entertaining language generation). So if there is a method that achieves the same goal with a model that is simple enough to be used by normal developers and researchers without OpenAI-scale infrastructure, that does seem buzz-worthy.

> So if there is a method that achieves the same goal with a model that is simple enough to be used by normal developers and researchers without OpenAI-scale infrastructure, that does seem buzz-worthy. That's trivial and has already been done. GPT-3 didn't even get SOTA on SuperGlue. These are the sort of misunderstandings that could have been avoided if the title was better. In general, paper with "new variation of…

> GPT-3 didn't even get SOTA on SuperGLUE.

Of course neither GPT-3 nor the PET paper claim SOTA on SuperGLUE. They used a few-shot learning setup with 32 examples per task The normal SuperGLUE setup has hundreds or thousands of examples per task [1].

> In general, paper with "new variation of cloze pre-training task for this specific task" is a new section of the literature that is rapidly becoming sort of mundane and uninteresting because there are so many papers doing small variations of the same basic idea.

Could you please link to some of the work you are referring to?

[1] Table 1 in https://w4ngatang.github.io/static/papers/superglue.pdf

Re: AI training method exceeds GPT-3 performance with fewer parameters

#69

Post by leereeves »

msamwald wrote: post #66
Veedrac wrote: post #50

Earlier quoted context omitted.

I had this discussion with Timo Schick, the first author, on Reddit. My final comment is copied below, with the relevant context. Note that GPT-3's approach to *GLUE involved no training on the task, just a good choice of prompt, whereas PET and iPET also use fine-tuning. Also, because distillation takes large amounts of training data, they use ensembles in the true few-shot regime, so their parameter efficiency is s…

A shorter reply would be: It would be great to compare PET not only to GPT-3, but also to other models, especially ones geared towards few-shot learning. Do you know of any other models that should be used for such a comparison, or are there already any relevant results on SuperGLUE that should be mentioned?

This appears to be SOTA on SuperGLUE with few-shot learning.

PET (well, a version called iPET from the same author) is at #9 on the SuperGLUE leaderboard [1], and none of the models above it mention being evaluated by few-shot learning.

1: https://super.gluebenchmark.com/leaderboard/

Re: AI training method exceeds GPT-3 performance with fewer parameters

#70

Post by whimsicalism »

msamwald wrote: post #68
whimsicalism wrote: post #64

Earlier quoted context omitted.

> So if there is a method that achieves the same goal with a model that is simple enough to be used by normal developers and researchers without OpenAI-scale infrastructure, that does seem buzz-worthy. That's trivial and has already been done. GPT-3 didn't even get SOTA on SuperGlue. These are the sort of misunderstandings that could have been avoided if the title was better. In general, paper with "new variation of…

> GPT-3 didn't even get SOTA on SuperGLUE. Of course neither GPT-3 nor the PET paper claim SOTA on SuperGLUE. They used a few-shot learning setup with 32 examples per task The normal SuperGLUE setup has hundreds or thousands of examples per task [1]. > In general, paper with "new variation of cloze pre-training task for this specific task" is a new section of the literature that is rapidly becoming sort of mundane an…

I was sloppy in my skimming of the paper - upon closer read it does actually seem quite different than that literature I mentioned (examples: RoBERTa, XLNet). I'll be reading it more carefully, but can now better understand the comparison to GPT-3.
Post reply on HN