Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: Language Models Are Few-Shot Learners

arxiv.org

Post reply on HN
Print view View original

141–150 of 212 posts

Re: GPT-3: Language Models Are Few-Shot Learners

#141

Post by nyolfen »

tanilama wrote: post #135

Just to point out, that text that feels most humanly generated from GPT-3, seems heavily paraphrasing from the following articles: https://www.washingtonpost.com/religion/2020/01/03/united-me... https://www.washingtonpost.com/archive/local/1985/09/07/unit... GPT-3: The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. WP: The church has lost 1.6 m…

that particular fact has probably been printed in more than one place with more than one phrasing; is there a reason you think it was drawn from that one in particular?

Re: GPT-3: Language Models Are Few-Shot Learners

#142

Post by runawaybottle »

maest wrote: post #114
wyattpeak wrote: post #78

Earlier quoted context omitted.

I don't know if it says something about text generation or human text processing, but whenever I read an example of computer generated text, all through I think "I can't tell this is machine generated, it seems completely natural," and the only giveaway is that at the end I have no idea what it said. It's a pretty eerie feeling. It's as though both the AI and my short-term processing only pay attention to a context o…

That would suggest combining 2 models: one to decide what the macro text structure should be and a different one (GPT) to decide how to fill in all the text flesh.

That takes the whole write drunk, edit sober to a whole new level.

GPT2 the drunken novelist.

Re: GPT-3: Language Models Are Few-Shot Learners

#143

Post by conistonwater »

sillysaurusx wrote: post #84
julianjm wrote: post #77

I am not a fan of this trend of "Language Models Are X" in recent work particularly out of OpenAI. I think it's a rhetorical sleight of hand which hurts the discourse. Like, the exact same paper could have instead been titled "Few-Shot Learning with a Large-Scale Language Model" or similar. But instead there seems to be this extremely strong desire to see certain ineffable qualities in neural networks. Like, it's a l…

But as a VERY publicly watched lab, they have a serious duty I was nodding right along with you, and then... OpenAI has no duty. It doesn't matter if they're publicly watched. What matters is whether the field of AI can be advanced, for some definition of "advanced" equal to "the world cares about it." It's important to let startups keep their spirit. Yeah, OpenAI is one of the big ones. DeepMind, Facebook AI, OpenAI…

> OpenAI has no duty. ...

Of course they do! It's the same duty as every scientist has in advancing the public understanding of science. You seem to be replying to OP as if they said that only big AI research groups this duty, but this is just not so. Furthermore, when a prominent group of scientists conduct themselves poorly, it is not enough to say that they have no special extra duty due to being famous, they already must communicate properly because they are scientists and part of the scientific community.

I think one reason these conversations get so muddled is because it's all new and really pretty cool, so it becomes hard to tell what's skepticism and what's naysaying.

> Such mindsets are quite effective in causing teams to slow down and second-guess themselves.

Absolutely not, this goes directly against the scientific method. Such "mindsets" of trying to make sure that your results are correct and accurately presented without embellishment are a cornerstone of science. Of course it causes them to slow down! They have more work to do! Second-guessing themselves and their experiments is the whole fucking point.

Re: GPT-3: Language Models Are Few-Shot Learners

#145

Post by specialist »

sillysaurusx wrote: post #84
julianjm wrote: post #77

I am not a fan of this trend of "Language Models Are X" in recent work particularly out of OpenAI. I think it's a rhetorical sleight of hand which hurts the discourse. Like, the exact same paper could have instead been titled "Few-Shot Learning with a Large-Scale Language Model" or similar. But instead there seems to be this extremely strong desire to see certain ineffable qualities in neural networks. Like, it's a l…

But as a VERY publicly watched lab, they have a serious duty I was nodding right along with you, and then... OpenAI has no duty. It doesn't matter if they're publicly watched. What matters is whether the field of AI can be advanced, for some definition of "advanced" equal to "the world cares about it." It's important to let startups keep their spirit. Yeah, OpenAI is one of the big ones. DeepMind, Facebook AI, OpenAI…

Well. OpenAI did have their own "Don't be evil" moment.

https://en.wikipedia.org/wiki/OpenAI#Motives

Given their prophylactic strategy, "AI for everyone", they could argue that hype generates public interest.

Re: GPT-3: Language Models Are Few-Shot Learners

#146

Post by taneq »

cs702 wrote: post #130

This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…

> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain

Wow, that is WAY closer than I thought we were.

Re: GPT-3: Language Models Are Few-Shot Learners

#147

Post by skdotdan »

cs702 wrote: post #130

This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…

AFAIK they used half-precision (Float16)

Re: GPT-3: Language Models Are Few-Shot Learners

#148

Post by runawaybottle »

MauranKilom wrote: post #140
ragebol wrote: post #110

Earlier quoted context omitted.

My thoughts exactly. It seems a person incapable of proper grammar (like a baby) has some concept or thought it wants to express, but can't because it doesn't know the words etc. These language models seem to know the words and the grammar etc, but lack a underlying concept they want to express. There are systems that derive 'thought-vectors', but I'd be interested going the other way: somehow create such a 'thought-…

I have been thinking about this kind of thing too. What if there was some way to feed your condensed thoughts into such a model and it writes a paper/blog post/article? Essentially, one should be able to use these models to "interpolate" the writing around the raw meaning/content. Typing assistance (think Grammarly) already allows you to refine finished writing to be more in line with what some language model expects…

If we get to large scale text generation like that, we are all going to have to become even better skimmers due to how the meta language will evolve.

So take your standard press release. We know about two thirds of it is just fluff. In other words, we are accepting the mass of fluff as one word in our language, it translates to ‘ignore’.

Our own language will change in that case.

Re: GPT-3: Language Models Are Few-Shot Learners

#149

Post by Veedrac »

lopmotr wrote: post #55
nullc wrote: post #50

10^4 petaflop-second/days. They missed an opportunity to be the first paper to measure their computation in mole flops.

Rather, chemists constantly miss opportunities to use actual numbers instead of their lazy legacy mole nonsense. Nobody really seems to use SI prefixes beyond peta or occasionally exa. But they could have called this 900 zetta-flop. (10^4 peta-flop/s-days)

A petaflop/s-day is a nice unit because it's roughly what you get from running the fastest AI accelerators for a day. The V100 they used is 0.13 petaflop/s FP16, for example, and the recently announced A100 ups that to 0.3.

Re: GPT-3: Language Models Are Few-Shot Learners

#150

Post by tanilama »

gas9S9zw3P9c wrote: post #136
tanilama wrote: post #135

Just to point out, that text that feels most humanly generated from GPT-3, seems heavily paraphrasing from the following articles: https://www.washingtonpost.com/religion/2020/01/03/united-me... https://www.washingtonpost.com/archive/local/1985/09/07/unit... GPT-3: The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. WP: The church has lost 1.6 m…

Where do you draw the line between "generating" and "copy-pasting from its vast memory"? Why do you think what humans do is not copy & pasting different snippets of information they have come across in the past? Isn't that what grammar is? A bunch of rules you've come across a lot of times? Other than the given prompt, the models don't have a goal. So what other than copying and adjusting would they do?

To evaluate generic model is hard.

For example, for image synthesis in GAN, the widely used Inception score balances between authenticity of the generate samples vs the variety as well, to make sure the model is not copy-pasting.

In this particular case, apparently the same event has been reported multiple times by different news agency. Even if the exact one are excluded, still it is suspicious how much less the model is being protected from knowing the subject itself.

An analogy would exam in real world. Often, some of the questions aren't leaked as is, but paraphrased yet stay close enough to the source.

In this particular case though, I disagree it is reaching human level generation. They can tested the model with an unseen events, which happen after the model is trained to test how well it generalize.

Post reply on HN