Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: Language Models Are Few-Shot Learners

arxiv.org

Post reply on HN
Print view View original

11–20 of 212 posts

Re: GPT-3: Language Models Are Few-Shot Learners

#11

Post by sdan »

Read though most of the paper and here's what GPT-3 is:

If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results.

With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example)

The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param model (relative to Grover/T5's 1.5/11B param models). This shows that even with these huge models, smaller models can outperform them in certain instances with lesser param models.

Also I think they did a good job with explaning the ethics and morals around what models like these mean / what biases this has.

Re: GPT-3: Language Models Are Few-Shot Learners

#13

Post by ericlewis »

sdan wrote: post #11

Read though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param mo…

Would you have any easy to explain insight in to how these perform better than larger models? I’ve always wanted to understand that as a technically adept and somewhat familiar (briefly) person who has explored what such models can do.

Re: GPT-3: Language Models Are Few-Shot Learners

#14

Post by sdan »

ericlewis wrote: post #13
sdan wrote: post #11

Read though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param mo…

Would you have any easy to explain insight in to how these perform better than larger models? I’ve always wanted to understand that as a technically adept and somewhat familiar (briefly) person who has explored what such models can do.

Throw more computers and do some model architecture changes

/s (although sometimes it's true)

Re: GPT-3: Language Models Are Few-Shot Learners

#15

Post by sacred_numbers »

Check out the poetry it generated in Figure F.1 (especially poem 4). I don't know how many bad poems the authors had to sort through to find these, but this AI is generating real poetry. If I didn't know they were computer generated I doubt I would have even considered that they didn't come from a human. This is a huge accomplishment and the team that created GPT-3 should be proud.

Re: GPT-3: Language Models Are Few-Shot Learners

#17

Post by anoncareer0212 »

judge2020 wrote: post #8
koolba wrote: post #6

What is this and why does it take the top two spots on HN?

One thread will (probably) be merged into the other, but GPT-2 was an extremely popular OpenAI project that generated long, realistic-sounding text/articles if you gave it a simple starting sentence or topic sentence. GPT-3 is an iteration on that, so it's likely a huge improvement.

It doesn't sound like it's an improvement at all, but instead requires less training data to produce worse results?

Re: GPT-3: Language Models Are Few-Shot Learners

#18

Post by lanternfish »

sacred_numbers wrote: post #15

Check out the poetry it generated in Figure F.1 (especially poem 4). I don't know how many bad poems the authors had to sort through to find these, but this AI is generating real poetry. If I didn't know they were computer generated I doubt I would have even considered that they didn't come from a human. This is a huge accomplishment and the team that created GPT-3 should be proud.

They claim the completions are "uncurated" which I would assume means that they didn't sort through any of them, and these were just off the top. That seems pretty impressive, so I wouldn't be surprised if I was incorrect.

Re: GPT-3: Language Models Are Few-Shot Learners

#19

Post by jointpdf »

sacred_numbers wrote: post #15

Check out the poetry it generated in Figure F.1 (especially poem 4). I don't know how many bad poems the authors had to sort through to find these, but this AI is generating real poetry. If I didn't know they were computer generated I doubt I would have even considered that they didn't come from a human. This is a huge accomplishment and the team that created GPT-3 should be proud.

The paper indicates that those poems were uncurated.
Post reply on HN