GPT-3: Language Models Are Few-Shot Learners • Hacker News Board

Re: GPT-3: Language Models Are Few-Shot Learners

#151

Post by nwienert » Fri, May 29, 2020, 2:14 PM UTC

longtom wrote: post #80

KhoomeiK wrote: post #69

Earlier quoted context omitted.

Is there any specific reasoning behind equating 1 synapse to 1 NN parameter? Seems a bit simplistic. Seems to me like a synapse probably has more computational ability than a single parameter.

Real neurons have many other trainable parameters and a lot more computational structure, so this is of course a simplifying assumption, but it is not entirely baseless either as it is known ANNs can approximate any function in theory, which may suggest synaptic weights do the heavy lifting in biological brains (since what more than general do you need?). Though biological brains are likely overly complicated due to…

The hydrocephalus story looks a bit sketchy [0].

Also if anything brains are hyper optimized for many things (based on the many specialized sub-units). I’d bet we are essentially not unsupervised, and the sub-units of the brain are essentially fine tuned for many tasks, and hyper optimized to use all their resources incredibly efficiently (memory optimization must be intense). Not that the generative models won’t get close in some general way relatively soon, but I could see human brains being another 10-1000x more powerful than your ballpark pretty easily.

[0] https://www.gwern.net/Hydrocephalus

Re: GPT-3: Language Models Are Few-Shot Learners

#152

Post by safog » Fri, May 29, 2020, 2:14 PM UTC

taneq wrote: post #146

cs702 wrote: post #130

This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…

> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain Wow, that is WAY closer than I thought we were.

I remember reading about fruit fly brains and how we're at a point where we can computationally simulate them now, but I'm not sure where that went.

Anyone know?

Re: GPT-3: Language Models Are Few-Shot Learners

#153

Post by sacred_numbers » Fri, May 29, 2020, 2:15 PM UTC

quotemstr wrote: post #58

sacred_numbers wrote: post #15

Check out the poetry it generated in Figure F.1 (especially poem 4). I don't know how many bad poems the authors had to sort through to find these, but this AI is generating real poetry. If I didn't know they were computer generated I doubt I would have even considered that they didn't come from a human. This is a huge accomplishment and the team that created GPT-3 should be proud.

To be fair, human poem authors generate and reject tons of bad poems too!

The difference is that the best authors know how to reject their own bad poems. I think the next step is a GAN that learns to weed out bad writing.

Re: GPT-3: Language Models Are Few-Shot Learners

#154

Post by ralls_ebfe » Fri, May 29, 2020, 2:18 PM UTC

minimaxir wrote: post #47

ganstyles wrote: post #37

Earlier quoted context omitted.

It took them a while to release GOT-2 full model because of the implications for things like spambots. The GPT-3 paper indicates that they have been monitoring forums and noticed that bad actors haven't really been using GPT-2 for their own devices. That's unsurprising because GPT-2 takes a lot of hardware to run and I assume it messes with the economics of spamming. GPT-3 will take significantly more resources to ru…

> That's unsurprising because GPT-2 takes a lot of hardware to run and I assume it messes with the economics of spamming. GPT-2 doesn't require as many resources to run as you would expect: even from the 1.5B model, you can mass-produce passing spam comments for less than a dollar an hour in GPU costs: https://docs.aitextgen.io/tutorials/generate_1_5b/ Pure text spam in general is less effective in 2020; it's content…

I am confused. By content you mean audio/visual content in contrast to textual content?

Re: GPT-3: Language Models Are Few-Shot Learners

#155

Post by ricardorivaldo » Fri, May 29, 2020, 2:20 PM UTC

safog wrote: post #152

taneq wrote: post #146

Earlier quoted context omitted.

> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain Wow, that is WAY closer than I thought we were.

I remember reading about fruit fly brains and how we're at a point where we can computationally simulate them now, but I'm not sure where that went. Anyone know?

Take a look at http://fruitflybrain.org/

Re: GPT-3: Language Models Are Few-Shot Learners

#156

Post by lowdose » Fri, May 29, 2020, 2:29 PM UTC

carlmr wrote: post #120

pasquinelli wrote: post #95

Earlier quoted context omitted.

To me it reads like a child telling a story, but that this child has an adult's ability to use language. When children tell a story they aren't going anywhere with it but don't know how to cover it up.

I know plenty of adults that can't seem to get to a point.

Kevin Hart on the Joe Rogan show comes to mind.

Re: GPT-3: Language Models Are Few-Shot Learners

#157

Post by sdbrown » Fri, May 29, 2020, 2:36 PM UTC

RoboTeddy wrote: post #68

Does it have a latent personality? How would it answer the questions on a 5-factor personality test? Would its results on the test be consistent with its behavior (generated text) in other situations?

Or, can it (like humans?) adapt its responses to suit the style of the question? Like if you start asking it lots of antagonizing questions, will it become more or less antagonistic itself?

Re: GPT-3: Language Models Are Few-Shot Learners

#158

Post by longtom » Fri, May 29, 2020, 2:43 PM UTC

nwienert wrote: post #151

longtom wrote: post #80

Earlier quoted context omitted.

Real neurons have many other trainable parameters and a lot more computational structure, so this is of course a simplifying assumption, but it is not entirely baseless either as it is known ANNs can approximate any function in theory, which may suggest synaptic weights do the heavy lifting in biological brains (since what more than general do you need?). Though biological brains are likely overly complicated due to…

The hydrocephalus story looks a bit sketchy [0]. Also if anything brains are hyper optimized for many things (based on the many specialized sub-units). I’d bet we are essentially not unsupervised, and the sub-units of the brain are essentially fine tuned for many tasks, and hyper optimized to use all their resources incredibly efficiently (memory optimization must be intense). Not that the generative models won’t get…

Thanks I was not aware of these details about the hydrocephalus story.

Re: GPT-3: Language Models Are Few-Shot Learners

#159

Post by alphagrep12345 » Fri, May 29, 2020, 2:48 PM UTC

cs702 wrote: post #130

This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…

What do you mean by text instructions? If I want to translate a sentence, would I just feed in model - translate "Hello world"?

Re: GPT-3: Language Models Are Few-Shot Learners

#160

Post by Traktor » Fri, May 29, 2020, 2:53 PM UTC

alphagrep12345 wrote: post #159

cs702 wrote: post #130

This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…

What do you mean by text instructions? If I want to translate a sentence, would I just feed in model - translate "Hello world"?

See page 7 of the paper. You give the model instruction such as "Translate from X to Y" then you pass examples (if you go for few shot) followed by the sentence you want to translate.