Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: Language Models Are Few-Shot Learners

arxiv.org

Post reply on HN
Print view View original

181–190 of 212 posts

Re: GPT-3: Language Models Are Few-Shot Learners

#181

Post by gwern »

gambler wrote: post #176
gwern wrote: post #169

Earlier quoted context omitted.

That's a very sloppy post. He does a single example, not even running locally or changing sampling parameters, and then concludes that GPT-2 is doing nothing but pattern-matching? A lot of people underestimate NNs because the sampling from them (top-k! how much dumber and cruder can you get? nucleus works better, but is still obviously suboptimal) destroys a lot of dark knowledge. I noticed this with Gary Marcus's cl…

The article I linked to makes claims only about the model it tests and since it actually links to an online implementation, anyone can try to reproduce the results and see for themselves. This is more than I can say about most chatter about ML. >Such tests can prove the presence of knowledge, but not the absence... This sounds like a setup for non-falsifiable beliefs.

> The article I linked to makes claims only about the model it tests and since it actually links to an online implementation, anyone can try to reproduce the results and see for themselves.

And I did (using my own local GPT-2-1.5b install which let me set the hyperparameters rather than restricting it to inappropriate hardwired ones of an online service), I linked to another person demonstrating the same thing, I pointed out the extensive GPT-3 evaluation OA did, and here, have another link about how bad querying of language models leads to highly misleading results about how much they know: https://arxiv.org/abs/1911.12543 Measurement error in general biases estimates towards zero.

> This sounds like a setup for non-falsifiable beliefs.

It's just as non-falsifiable as, say, concepts like 'lower bounds' or 'bugs'.

Re: GPT-3: Language Models Are Few-Shot Learners

#182

Post by gwern »

sillysaurusx wrote: post #96
runawaybottle wrote: post #9

Earlier quoted context omitted.

Dude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment. We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles. It’s over, we lost. Morpheus: What if I told you that, throughout your whole life,…

Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…

You are missing the point of the paper about few-shot learning. That's the entire paper: just doing new untrained task after task. The entire point of the paper is that you can 'reprogram' GPT-3 to do just about anything just by stuffing its context with examples, and it'll pick up brandnew entities or words or concepts just by examples (see the examples of defining novel gibberish words and asking GPT-3 to use them in a sentence - it does so. it "learned" new words by reading the examples, understanding, and propagating them through the 'fast weights' of self-attention, even though its 'slow weights' are fixed). Now, if GPT-3 can do that already so well, sometimes hitting SOTA on untrained tasks purely by internal meta-learning without changing its weights, what would a 10-trillion parameter model do? Or one with recurrency like XL or Compressive? How much training do you really need if the few-shot learning capabilities are so great you can make it do countless tasks just by providing examples or descriptions in the prompt.

Re: GPT-3: Language Models Are Few-Shot Learners

#183

Post by gwern »

drdeca wrote: post #99
sdan wrote: post #11

Read though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param mo…

Have you tried making GPT2 do zero-shot poetry writing? It's not great at it, but it is good enough at it to get something interesting enough if you try a number of times. Go to talktotransformer.com/ and give it the prompt "Here is a poem I wrote:" or "Here is my favorite poem:" . I'm sure GPT3 would produce much better and more consistent results, but GPT2 will produce something that looks generally like a poem fre…

Yes, it can but GPT-2-1.5b isn't too great at it. What really struck me looking at the examples is that the random GPT-3 poem samples are practically as good as my GPT-2-1.5b finetuned on hundreds of megabytes of poetry at considerable effort & expense: https://www.gwern.net/GPT-2 That's... both really awesome and dispiriting.

Re: GPT-3: Language Models Are Few-Shot Learners

#185

Post by drusepth »

freeone3000 wrote: post #48
anoncareer0212 wrote: post #17

Earlier quoted context omitted.

It doesn't sound like it's an improvement at all, but instead requires less training data to produce worse results?

MUCH less training for SLIGHTLY worse results. It's a huge benefit to be able to make this trade-off.

Is the reverse also true? If you have the training data necessary for "good" results on GPT-2, is it generally correct to assume that it would provide better results on your task than GPT-3?

Re: GPT-3: Language Models Are Few-Shot Learners

#186

Post by ericlewis »

Analog24 wrote: post #67
ericlewis wrote: post #13

Earlier quoted context omitted.

Would you have any easy to explain insight in to how these perform better than larger models? I’ve always wanted to understand that as a technically adept and somewhat familiar (briefly) person who has explored what such models can do.

The key insight in this paper is that the new (larger) model was not "fine-tuned" on the downstream NLP tasks. In other words, after it's trained on unsupervised (you could call it self-supervised in this case) data to do simple things like predict the next word (hence why it doesn't take any real supervision) it can then be used to do very specific tasks like answer questions or translating text without further supe…

very cool, thanks for explaining!

Re: GPT-3: Language Models Are Few-Shot Learners

#188

Post by longtom »

p1esk wrote: post #66
longtom wrote: post #38

Earlier quoted context omitted.

It seems it has (rudimentarily) learned concepts general enough to be logic itself. That is general intelligence. Now hook it up to reinforcement circuitry and make it even larger and it will mark the end to life as we know it. GTP-3 has 175 billion parameters, but the human brain has 100 trillion synapses, so 0.175%. NN model capacity currently has a 3.4 month doubling time.[1] In 7-10 doublings we'll be in a simila…

How many of those 100T synapses are dedicated to language skills?

This is an interesting question. Likely more than 0.1%, perhaps 20-40% I'd guess. Which would be the lower estimate I provided.

Re: GPT-3: Language Models Are Few-Shot Learners

#189

Post by The_rationalist »

sillysaurusx wrote: post #96
runawaybottle wrote: post #9

Earlier quoted context omitted.

Dude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment. We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles. It’s over, we lost. Morpheus: What if I told you that, throughout your whole life,…

Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…

About learning, the ERNIE language model has continuous learning meaning it has not the necessity to fully retrain (I believe)

Also GPT 3 is obscoleted by order of magnitudes by SMIM https://arxiv.org/abs/2003.02645

Re: GPT-3: Language Models Are Few-Shot Learners

#190

Post by The_rationalist »

GPT 3 is obscoleted by order of magnitudes. SMIM has achieved 4.6 of perplexity vs 20 for GPT 3 with and with a thousand less parameters https://arxiv.org/abs/2003.02645 This is the breakthrough of the year and will be silent until the few nerds like me propagate the news to the mainstream
Post reply on HN