11–20 of 82 posts
Earlier quoted context omitted.
Yes, it can do simple yes/no questions but can it write dad jokes like GPT-3?
For me another question is crucial: is it open (i.e. is there an open source reference implementation) or closed like the so-called "OpenAI" products/services?
Semi-supervised training is always interesting, but leading with fact that it outperforms GPT-3 on superGLUE in the few shot setting where GPT-3 isn't training/fine-tuning but PET is.
To make it clear: they finetune PET using synthetic data and then compare the results of this to GPT-3 where GPT-3 is initalized to prepare for querying.
I would like to see how much overlap the synthetic data has with the queries used.
GPT3's strength is on language generation, so using *GLUE for evaluating it (where encoder type models are just better) and claiming to have 99.9% less parameters is sensationalism.
Few-shot GPT3 outperforms a BERT-based baseline.
I'm not surprised, a space with 175B parameters will be very sparse.
If you have a really big compute cluster, it makes sense to do experiments like this. It would be foolish to constantly try new methods without occasionally checking to see how far you can push current methods.
A similar thing happened with VGGnet in image classification. It achieved SoTA with a huge amount of parameters, using the standard techniques of the time, but increasing the network depth. Later, people discovered a lot of tricks to get similar accuracy with fewer parameters.
I just wish I had the vocabulary to truly appreciate what's going on here.
GPT3's strength is on language generation, so using *GLUE for evaluating it (where encoder type models are just better) and claiming to have 99.9% less parameters is sensationalism.
This isn't strictly true. This is comparing to "GPT3 as a few-shot learner"[1] as opposed to the fine tuned models that everyone else use. Few-shot GPT3 outperforms a BERT-based baseline. [1] https://github.com/openai/gpt-3