Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: Language Models Are Few-Shot Learners

arxiv.org

Post reply on HN
Print view View original

31–40 of 212 posts

Re: GPT-3: Language Models Are Few-Shot Learners

#31

Post by aquajet »

whymauri wrote: post #27
robkop wrote: post #23

Earlier quoted context omitted.

I would hazard a guess that they will release versions that will be a smaller size (they have in the past). But in order to run this, you'd just have to use a cloud provider, first guesses say it'll be 500GB+ of just weights that ideally you want in memory.

Could we bank on the Lottery Ticket Hypothesis, distillation, or other model compression algorithms to make these models smaller?

I would guess so, but compressing it by 1/3rd it's size (ie. distilgpt) would still be quite large. To be fair, I don't know if distillation scales like that.

Re: GPT-3: Language Models Are Few-Shot Learners

#32

Post by ganstyles »

aquajet wrote: post #26
ganstyles wrote: post #24

Earlier quoted context omitted.

This post may be tongue in cheek, but realistically that's right. I ran the Facebook chatbot that was posted maybe a month ago and didn't get good performance until I was using many Tesla v100 GPUs, which are $8k each. Thankfully modern CSPs like Azure allow renting rigs that can handle this pretty easily.

How much does that cost?

The largest AWS instances are $30-something/hr on demand, less with reserved time. So pretty expensive.

Re: GPT-3: Language Models Are Few-Shot Learners

#33

Post by Reelin »

anoncareer0212 wrote: post #17
judge2020 wrote: post #8

Earlier quoted context omitted.

One thread will (probably) be merged into the other, but GPT-2 was an extremely popular OpenAI project that generated long, realistic-sounding text/articles if you gave it a simple starting sentence or topic sentence. GPT-3 is an iteration on that, so it's likely a huge improvement.

It doesn't sound like it's an improvement at all, but instead requires less training data to produce worse results?

This is a massive improvement to the extent that previously you had to retrain (ie update) the stock model on a specialized dataset to get good results for a particular task.

Re: GPT-3: Language Models Are Few-Shot Learners

#34

Post by scottlocklin »

ericlewis wrote: post #13
sdan wrote: post #11

Read though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param mo…

Would you have any easy to explain insight in to how these perform better than larger models? I’ve always wanted to understand that as a technically adept and somewhat familiar (briefly) person who has explored what such models can do.

>how these perform better than larger models

they probably don't particularly; their inventors seem to excel in their PR budget rather than their verifiable innovations

Re: GPT-3: Language Models Are Few-Shot Learners

#35

Post by changoplatanero »

spats1990 wrote: post #7

Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am…

when i worked on text generation I had a hack where I wouldn't let the model generate the same trigram more than twice. It seems like that hack could have improved this example.

Re: GPT-3: Language Models Are Few-Shot Learners

#36

Post by frutiger »

leesec wrote: post #28

This part really freaked me out... GPT-2 couldn't do math: Context → Passage: Saint Jean de Br´ebeuf was a French Jesuit missionary who travelled to New France in 1625. There he worked primarily with the Huron for the rest of his life, except for a few years in France from 1629 to 1633. He learned their language and culture, writing extensively about each to aid other missionaries. In 1649, Br´ebeuf and another missi…

This is not my idea of math.

Re: GPT-3: Language Models Are Few-Shot Learners

#37

Post by ganstyles »

gre wrote: post #10

Is it sentient yet? /s Real question, are they going to release the full model?

It took them a while to release GOT-2 full model because of the implications for things like spambots. The GPT-3 paper indicates that they have been monitoring forums and noticed that bad actors haven't really been using GPT-2 for their own devices. That's unsurprising because GPT-2 takes a lot of hardware to run and I assume it messes with the economics of spamming.

GPT-3 will take significantly more resources to run. However, part of me doesn't want it released ever because of the implications of what bad actors could do with it.

Re: GPT-3: Language Models Are Few-Shot Learners

#38

Post by longtom »

leesec wrote: post #28

This part really freaked me out... GPT-2 couldn't do math: Context → Passage: Saint Jean de Br´ebeuf was a French Jesuit missionary who travelled to New France in 1625. There he worked primarily with the Huron for the rest of his life, except for a few years in France from 1629 to 1633. He learned their language and culture, writing extensively about each to aid other missionaries. In 1649, Br´ebeuf and another missi…

It seems it has (rudimentarily) learned concepts general enough to be logic itself. That is general intelligence. Now hook it up to reinforcement circuitry and make it even larger and it will mark the end to life as we know it.

GTP-3 has 175 billion parameters, but the human brain has 100 trillion synapses, so 0.175%. NN model capacity currently has a 3.4 month doubling time.[1] In 7-10 doublings we'll be in a similar ballpark, i.e. 2-3 years.

[1] https://openai.com/blog/ai-and-compute/

Re: GPT-3: Language Models Are Few-Shot Learners

#39

Post by jointpdf »

jointpdf wrote: post #19
sacred_numbers wrote: post #15

Check out the poetry it generated in Figure F.1 (especially poem 4). I don't know how many bad poems the authors had to sort through to find these, but this AI is generating real poetry. If I didn't know they were computer generated I doubt I would have even considered that they didn't come from a human. This is a huge accomplishment and the team that created GPT-3 should be proud.

The paper indicates that those poems were uncurated.

Relatedly, here is a sample of uncurated and unconditional (all “topics”—not just poetry): https://raw.githubusercontent.com/openai/gpt-3/master/175b_s...

Scanning through these, the text seems significantly less zany than the random GPT-2 samples. It’s genuinely difficult to spot the signs that these were generated, even with the knowledge that they were.

Post reply on HN