Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: Language Models Are Few-Shot Learners

arxiv.org

Post reply on HN
Print view View original

111–120 of 212 posts

Re: GPT-3: Language Models Are Few-Shot Learners

#111

Post by flippinburgers »

spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

Quite honestly the level of logical consistency in this generated text doesn't differ that much, I feel, from a huge swath of the kind of comments that I see on the internet. It may well be that there are already too many bots on the net to really understand what typical humans would be saying anyway, but I feel like this is already good enough for short form trickery.

I was so wrong about the internet. It is just going to become a landscape of garbage opinions and commentary on a larger and larger basis.

Re: GPT-3: Language Models Are Few-Shot Learners

#112

Post by stared »

drdeca wrote: post #99
sdan wrote: post #11

Read though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param mo…

Have you tried making GPT2 do zero-shot poetry writing? It's not great at it, but it is good enough at it to get something interesting enough if you try a number of times. Go to talktotransformer.com/ and give it the prompt "Here is a poem I wrote:" or "Here is my favorite poem:" . I'm sure GPT3 would produce much better and more consistent results, but GPT2 will produce something that looks generally like a poem fre…

Not for poems, but for AI-generated cities (inspired by "Invisible Cities" by Italo Calvino), e.g.:

PROMPT:

Cities & Lights

When you enter the city of Singapore during the night, you see lights: colorful and ubiquitous. Lights on every building, on every fountain, and in every park.

GENERATED:

Lights shining in a city in which the majority of people are now using mobile phones. Singapore has a bright future as a technology hub, and it 's not too late to make it happen.

...

On the other occasions I seeded with ~two sentences of "Invisible cities", and it worked like a charm, no fine-tuning.

Re: GPT-3: Language Models Are Few-Shot Learners

#113

Post by fnord123 »

Nursie wrote: post #107
wyattpeak wrote: post #78

Earlier quoted context omitted.

I don't know if it says something about text generation or human text processing, but whenever I read an example of computer generated text, all through I think "I can't tell this is machine generated, it seems completely natural," and the only giveaway is that at the end I have no idea what it said. It's a pretty eerie feeling. It's as though both the AI and my short-term processing only pay attention to a context o…

> the only giveaway is that at the end I have no idea what it said So it's a lot like corporate executive speak then? I agree with your point, it does seem very much like valid speech, but somehow the informational content is missing. It's like speech without the actual comminication part.

I've read a lot of academic papers like this.

Re: GPT-3: Language Models Are Few-Shot Learners

#114

Post by maest »

wyattpeak wrote: post #78
spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

I don't know if it says something about text generation or human text processing, but whenever I read an example of computer generated text, all through I think "I can't tell this is machine generated, it seems completely natural," and the only giveaway is that at the end I have no idea what it said. It's a pretty eerie feeling. It's as though both the AI and my short-term processing only pay attention to a context o…

That would suggest combining 2 models: one to decide what the macro text structure should be and a different one (GPT) to decide how to fill in all the text flesh.

Re: GPT-3: Language Models Are Few-Shot Learners

#115

Post by kriro »

sillysaurusx wrote: post #96
runawaybottle wrote: post #9

Earlier quoted context omitted.

Dude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment. We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles. It’s over, we lost. Morpheus: What if I told you that, throughout your whole life,…

Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…

Very interesting comment, thanks for taking the time to write it :)

I think if memory is the only problem than optimizing training time should be more of a concern. I'm imagining a huge language model than can retrain very quickly. So I suppose it might be a decent idea to not measure it by perplexity or some human judgement score or whatever but rather by that score per compute units used.

Or in other words...maybe a bot that scores 90% on the fool a human scale and takes 1 day to compute from scratch is actually a lot less impressive than one that fools 70% but computes from scratch in 5 minutes.

And something "like Github for bot-memory" would be a pretty amazing tool. Roll back to some memory status and recompute with new data from there, branch for different datasets that represent different ways of interpreting the world etc.

Conceptually I like the idea of one "base model" that represents language and many different context models on top of it (finetuning the core model). Then some other subsystem that identifies the context and switches to that. I suppose each conversation could also be considered a mini-dataset.

Re: GPT-3: Language Models Are Few-Shot Learners

#116

Post by disgruntledphd2 »

sdan wrote: post #83
julianjm wrote: post #77

I am not a fan of this trend of "Language Models Are X" in recent work particularly out of OpenAI. I think it's a rhetorical sleight of hand which hurts the discourse. Like, the exact same paper could have instead been titled "Few-Shot Learning with a Large-Scale Language Model" or similar. But instead there seems to be this extremely strong desire to see certain ineffable qualities in neural networks. Like, it's a l…

Yeah its kind of odd why OpenAI makes these weird titles. "Few-Shot Learning with a Large-Scale Language Model" makes more sense. Even with their robot hand paper, they titled it along the lines of "we solved a rubrix cube" not "a robot hand manipulated the cube and solved it"

It's academic PR. Many disciplines have really cute titles that don't really match the research. My old discipline (psychology) was really known for this.

Re: GPT-3: Language Models Are Few-Shot Learners

#117

Post by disgruntledphd2 »

RoboTeddy wrote: post #68

Does it have a latent personality? How would it answer the questions on a 5-factor personality test? Would its results on the test be consistent with its behavior (generated text) in other situations?

That is a really interesting question.

Re: GPT-3: Language Models Are Few-Shot Learners

#118

Post by CardenB »

wyattpeak wrote: post #78
spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

I don't know if it says something about text generation or human text processing, but whenever I read an example of computer generated text, all through I think "I can't tell this is machine generated, it seems completely natural," and the only giveaway is that at the end I have no idea what it said. It's a pretty eerie feeling. It's as though both the AI and my short-term processing only pay attention to a context o…

This is because our current methods of machine learning are really good at finding patterns but not very good at enforcing underlying models.

Re: GPT-3: Language Models Are Few-Shot Learners

#120

Post by carlmr »

pasquinelli wrote: post #95
wyattpeak wrote: post #78

Earlier quoted context omitted.

I don't know if it says something about text generation or human text processing, but whenever I read an example of computer generated text, all through I think "I can't tell this is machine generated, it seems completely natural," and the only giveaway is that at the end I have no idea what it said. It's a pretty eerie feeling. It's as though both the AI and my short-term processing only pay attention to a context o…

To me it reads like a child telling a story, but that this child has an adult's ability to use language. When children tell a story they aren't going anywhere with it but don't know how to cover it up.

I know plenty of adults that can't seem to get to a point.
Post reply on HN