Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: Language Models Are Few-Shot Learners

arxiv.org

Post reply on HN
Print view View original

131–140 of 212 posts

Re: GPT-3: Language Models Are Few-Shot Learners

#131

Post by Udik »

wyattpeak wrote: post #78
spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

I don't know if it says something about text generation or human text processing, but whenever I read an example of computer generated text, all through I think "I can't tell this is machine generated, it seems completely natural," and the only giveaway is that at the end I have no idea what it said. It's a pretty eerie feeling. It's as though both the AI and my short-term processing only pay attention to a context o…

The example above is impressive because it actually makes sense, except for the last sentence of the first paragraph: "But this time, his publicist is saying he'll be wearing a tux no matter what."

Remove that, and there is a typical if completely uninteresting celebrity argument: one has to play eccentric in public occasions (he did it last year, he's planning to do it again) and the other chides him for what she feels it's maybe a lack of respect? And he replies that despite his best intentions he can't go against his conscience. There, done. It's a perfect little piece ready to be served in some celebrity gossip magazine.

Re: GPT-3: Language Models Are Few-Shot Learners

#132

Post by specialist »

sillysaurusx wrote: post #96
runawaybottle wrote: post #9

Earlier quoted context omitted.

Dude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment. We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles. It’s over, we lost. Morpheus: What if I told you that, throughout your whole life,…

Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…

I really like your observation about memory.

Because you seem open minded to wild ass guesses and going meta:

I have a hunch that general intelligence will be the ability to learn from mistakes. Not just optimization. I mean applying the scientific method.

Hypothesis, prediction, run experiment, compare expected vs actual. And having a notion, any notion, to explain the delta between expected and actual.

Am total noob about AI, philosophy, cognition. Don't know if anyone else is framing AGI this way. I could just be repeating something I heard.

Re: GPT-3: Language Models Are Few-Shot Learners

#133

Post by rydre »

sillysaurusx wrote: post #96
runawaybottle wrote: post #9

Earlier quoted context omitted.

Dude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment. We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles. It’s over, we lost. Morpheus: What if I told you that, throughout your whole life,…

Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…

Can't tell if it's human or GPT-2 tbh, the sentences are 'hard' to understand... like sort of un-naturally written, or translated from a foreign language using google translate or something.

Re: GPT-3: Language Models Are Few-Shot Learners

#134

Post by mannykannot »

nafey wrote: post #63
spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

What is it about ai generated texts that on skimming through it it makes sense, but if you try to slow down and understand it feels absurd and surreal.

Because language is being treated as a thing complete in itself, as opposed to being related to an external world?

One of the issues in the 'Limitations' section was a difficulty with "common-sense physics", such as with the question "if I put cheese into the fridge, will it melt?"

To answer that question, you have to ask the right questions, such as "what is a fridge?" "what is a fridge for?" "What does it mean for cheese to melt?" "what is the cause of cheese melting?" Then one should consider the follow-on questions, such as "what are typical fridge temperatures?" "what are typical cheese melting points?" "what temperature is the cheese likely to be at initially?" (at which point, it helps to introduce the concept of room temperature, and note that it typically falls between the other two.) From facts such as the answers to these questions, one can deduce the probable outcome of putting cheese in a refrigerator, but none of the answers so far explicitly state it.

Is it plausible that any learning, solely from the structure of and correlations between examples of language use, could develop the sort of analytical/modeling approach that I have just outlined? Instinctively, I don't find it very plausible, but I am not very certain in that view.

Re: GPT-3: Language Models Are Few-Shot Learners

#135

Post by tanilama »

Just to point out, that text that feels most humanly generated from GPT-3, seems heavily paraphrasing from the following articles:

https://www.washingtonpost.com/religion/2020/01/03/united-me...

https://www.washingtonpost.com/archive/local/1985/09/07/unit...

GPT-3:

The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church.

WP:

The church has lost 1.6 million members since 1968, when the Methodist Church merged with the considerably smaller Evangelical United Brethren to form the present United Methodist Church.

I think this model is still very impressive, the parameter itself speaks. But for this particular evaluation, the same news article may be removed from the training set, other news article that paraphrases the same story might not. IMO, the leakage still exists, it is hard to tell whether this model are really 'generating', or just copy-pasting from its vast memory.

Re: GPT-3: Language Models Are Few-Shot Learners

#136

Post by gas9S9zw3P9c »

tanilama wrote: post #135

Just to point out, that text that feels most humanly generated from GPT-3, seems heavily paraphrasing from the following articles: https://www.washingtonpost.com/religion/2020/01/03/united-me... https://www.washingtonpost.com/archive/local/1985/09/07/unit... GPT-3: The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. WP: The church has lost 1.6 m…

Where do you draw the line between "generating" and "copy-pasting from its vast memory"? Why do you think what humans do is not copy & pasting different snippets of information they have come across in the past? Isn't that what grammar is? A bunch of rules you've come across a lot of times?

Other than the given prompt, the models don't have a goal. So what other than copying and adjusting would they do?

Re: GPT-3: Language Models Are Few-Shot Learners

#137

Post by MauranKilom »

rydre wrote: post #133
sillysaurusx wrote: post #96

Earlier quoted context omitted.

Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…

Can't tell if it's human or GPT-2 tbh, the sentences are 'hard' to understand... like sort of un-naturally written, or translated from a foreign language using google translate or something.

Are you talking about the post you're replying to? Because I don't see those aspects in it at all...

Re: GPT-3: Language Models Are Few-Shot Learners

#138

Post by [deleted] »

spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

[deleted]

Re: GPT-3: Language Models Are Few-Shot Learners

#139

Post by xamdam »

wyattpeak wrote: post #78
spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

I don't know if it says something about text generation or human text processing, but whenever I read an example of computer generated text, all through I think "I can't tell this is machine generated, it seems completely natural," and the only giveaway is that at the end I have no idea what it said. It's a pretty eerie feeling. It's as though both the AI and my short-term processing only pay attention to a context o…

Relevant post about GPT2: https://srconstantin.wordpress.com/2019/02/25/humans-who-are...

Re: GPT-3: Language Models Are Few-Shot Learners

#140

Post by MauranKilom »

ragebol wrote: post #110
KhoomeiK wrote: post #73

Earlier quoted context omitted.

Yup, definitely different from how humans learn. A baby's speech would be almost the complete opposite, grammatically incorrect here and there but constructing a coherent line of thought for the most part.

My thoughts exactly. It seems a person incapable of proper grammar (like a baby) has some concept or thought it wants to express, but can't because it doesn't know the words etc. These language models seem to know the words and the grammar etc, but lack a underlying concept they want to express. There are systems that derive 'thought-vectors', but I'd be interested going the other way: somehow create such a 'thought-…

I have been thinking about this kind of thing too. What if there was some way to feed your condensed thoughts into such a model and it writes a paper/blog post/article?

Essentially, one should be able to use these models to "interpolate" the writing around the raw meaning/content. Typing assistance (think Grammarly) already allows you to refine finished writing to be more in line with what some language model expects, but imagine if it actually generated most of the text for you, based on small bites and chunks you throw at it.

Post reply on HN