GPT-3: Language Models Are Few-Shot Learners
1–10 of 212 posts
Re: GPT-3: Language Models Are Few-Shot Learners
#2paper: https://arxiv.org/abs/2005.14165
abstract:
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
Re: GPT-3: Language Models Are Few-Shot Learners
#3Duplicate thread at https://news.ycombinator.com/item?id=23345449 (with github link)
Re: GPT-3: Language Models Are Few-Shot Learners
#4The models have not yet been released, and it looks like someone has already asked about it in the issues: https://github.com/openai/gpt-3/issues/1
Re: GPT-3: Language Models Are Few-Shot Learners
#5Nice to see Jared Kaplan branching out into ML. He did fundamental work on CFTs/bootstrap in physics.
Re: GPT-3: Language Models Are Few-Shot Learners
#6What is this and why does it take the top two spots on HN?
Re: GPT-3: Language Models Are Few-Shot Learners
#7Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot:
Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm
Subtitle: Joaquin Phoenix pledged to not change for each awards event
Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am a shape-shifter. I can't change the world. I can only change myself." It was a promise to not change to fit into the Hollywood mold: "I think that's a really special thing, to not change yourself. I think it's a really special thing to say, `This is what's inside of me, I'm proud of it, and I'm not going to be ashamed because of the way that someone else thinks I should be.'" Now, it's the Oscars, and Phoenix is at it again. But this time, his publicist is saying he'll be wearing a tux no matter what.
Megyn Kelly was not impressed, and she let him have it on The Tonight Show. "You know, I feel like, I feel like you could have worn the tux," she says. "But you're saying you're a shape-shifter. I don't know if you can change your tux, but you can change your mind. You can change your mind. You can change your mind." Phoenix says he did, but it didn't stick. "I was like, `Okay, I'm going to wear a tuxedo to this thing.' And then I thought, `I don't want to wear a tuxedo to this thing.'" Kelly goes on to encourage him to change his mind again, but Phoenix says it's too late: "I'm committed to wearing this."
Re: GPT-3: Language Models Are Few-Shot Learners
#8What is this and why does it take the top two spots on HN?
One thread will (probably) be merged into the other, but GPT-2 was an extremely popular OpenAI project that generated long, realistic-sounding text/articles if you gave it a simple starting sentence or topic sentence. GPT-3 is an iteration on that, so it's likely a huge improvement.
Re: GPT-3: Language Models Are Few-Shot Learners
#9Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…
Dude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment.
We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles.
It’s over, we lost.
Morpheus: What if I told you that, throughout your whole life, you have been reading auto generated content?
Re: GPT-3: Language Models Are Few-Shot Learners
#10Is it sentient yet? /s
Real question, are they going to release the full model?