You do better than gpt3 with fewer parameters if you can have the same or better performances over the amount of tasks gpt3 can do well. To test a general model like gpt3 against something that can address a single problem does not make sense.
AI training method exceeds GPT-3 performance with fewer parameters
21–30 of 82 posts
Re: AI training method exceeds GPT-3 performance with fewer parameters
#22Sure they outperformed GTP-3... but can they DDoS my twitter account via a bunch of software engineers that won't stop talking about their model???
... even more, is the new model "so goddamn powerful" that it can't even be trusted with normal humans, and must be doled out (for money) a little bit at a time so that some normie doesn't accidentally create skynet?
Re: AI training method exceeds GPT-3 performance with fewer parameters
#23Slightly OT: Will GPT-3 itself will have any impact IRL? I ask because of its lacking options/competition on the hosting side (there's just Azure). This implies high prices hindering most use cases to break even/be profitable.
Re: AI training method exceeds GPT-3 performance with fewer parameters
#24I'm not surprised, a space with 175B parameters will be very sparse.
It's not obvious to me that this is the case. Chris Olah and others talk about "superposition" as a mechanism to explain "polysemantic" neurons that arise in image classifiers. To me, that suggests (using very vague, hand-wavy terms) that the optimization process is attempting to pack in as many concepts as possible into the finite parameter space. Certainly the scaling of GPT-3 suggests that these larger models are…
That is fascinating, thanks for the link.
Re: AI training method exceeds GPT-3 performance with fewer parameters
#25Good, "Open-AI" can suck it, the pricks.
Re: AI training method exceeds GPT-3 performance with fewer parameters
#26This title is somewhat misleading, it’s more that you can use semi supervised learning to improve your fine tuning on a small model that’s already been pretrained.
I think it’s impressive that this technique beats gpt-3 in any setting with such a small model, but for example, what stops you from applying this to gpt-3 itself and getting an equivalent gain?
Saying that it exceeds gpt-3 performance sounds like it does better during pretraining, which would be extremely impressive.
Re: AI training method exceeds GPT-3 performance with fewer parameters
#27We should not be surprised to see new language models achieving GPT-3 performance with fewer parameters. The purpose (I assume, as an outsider) of the GPT-* project is to try to find the upper bound of how good a language model can be, without caring too much about efficiency. GPT-3 essentially scaled up the known good architecture of GPT-2. If you have a really big compute cluster, it makes sense to do experiments l…
> A similar thing happened with VGGnet in image classification
I know I am at risk of a "Nobody will ever need more than 640KB of memory" comment, but the model size has exploded far in excess of our ability to improve GPU cards.
GPT-3 is two orders of magnitude larger than VGGnet-16. Back when the VGG paper was published a Titan Z (2014 gen) had 12gb of RAM, while a Titan RTX (current/previous gen) has just doubled to 24gb.
Re: AI training method exceeds GPT-3 performance with fewer parameters
#28GPT3's strength is on language generation, so using *GLUE for evaluating it (where encoder type models are just better) and claiming to have 99.9% less parameters is sensationalism.
That's kind of like saying that this super cheap SUV from 2003 is a better off-roader than the latest Ferrari. Like, true statement, but vacuous none the less.
Oh I love me a great analogy!!
Re: AI training method exceeds GPT-3 performance with fewer parameters
#29We should not be surprised to see new language models achieving GPT-3 performance with fewer parameters. The purpose (I assume, as an outsider) of the GPT-* project is to try to find the upper bound of how good a language model can be, without caring too much about efficiency. GPT-3 essentially scaled up the known good architecture of GPT-2. If you have a really big compute cluster, it makes sense to do experiments l…
> A similar thing happened with VGGnet in image classification I know I am at risk of a "Nobody will ever need more than 640KB of memory" comment, but the model size has exploded far in excess of our ability to improve GPU cards. GPT-3 is two orders of magnitude larger than VGGnet-16. Back when the VGG paper was published a Titan Z (2014 gen) had 12gb of RAM, while a Titan RTX (current/previous gen) has just doubled…
Your point is still valid but yesterday I learned about the newest workstation cards from nvidia - A6000 series with 48GB
https://www.engadget.com/nvidia-rtx-a6000-a40-gpu-profession...
Re: AI training method exceeds GPT-3 performance with fewer parameters
#30We should not be surprised to see new language models achieving GPT-3 performance with fewer parameters. The purpose (I assume, as an outsider) of the GPT-* project is to try to find the upper bound of how good a language model can be, without caring too much about efficiency. GPT-3 essentially scaled up the known good architecture of GPT-2. If you have a really big compute cluster, it makes sense to do experiments l…
> A similar thing happened with VGGnet in image classification I know I am at risk of a "Nobody will ever need more than 640KB of memory" comment, but the model size has exploded far in excess of our ability to improve GPU cards. GPT-3 is two orders of magnitude larger than VGGnet-16. Back when the VGG paper was published a Titan Z (2014 gen) had 12gb of RAM, while a Titan RTX (current/previous gen) has just doubled…
We can now each the performance of original VGG16/VGG19 with models that are much smaller. On ImageNet MobileNetsV2 matches the performance with 1/30 the size of weights, and same order of magnitude reduction in inference time and RAM usage.
The same will probably happen with GPT-3.
Of course we will probably also get models that surpass GPT-3 at the cost of even larger and more expensive models. The two are not at all exclusive.