Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's GPT-3 may be the biggest thing since Bitcoin

maraoz.com

Post reply on HN
Print view View original

41–50 of 554 posts

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#41

Post by teruakohatu »

nbardy wrote: post #9
minimaxir wrote: post #4

I published a response today to the sudden hype urging people to temper their expectations for GPT-3 a bit: https://minimaxir.com/2020/07/gpt3-expectations/ GPT-3 is objectively a step forward in the field of AI text-generation, but the current hype on VC Twitter misrepresents the model's current capabilities. GPT-3 isn't magic.

One of the biggest issues is with cherry-picking. Generative ML results benefit greatly from humans sampling the best results. They are capable of producing astonishing results but don’t do this consistently this has a huge impact on any effort to productize. For example I’ve seen quite a few examples of text->design, text->code, with GPT-3 you could build a demo in a day, but the product will probably be useless if…

I don't know about GPT-3 but playing around with GPT-2 I often got the impression that it was regurgitating learned knowledge (reddit comments) rather than actually coming up with something novel.

With so many weights, it practically encodes a massive Internet text database.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#42

Post by freeone3000 »

Still has some repetition and simplicity issues - article reads like it's written for an "eighth grade" reading level, and tends to repeat major points multiple times per line. There's also a lot less pronoun use than I'd expect in free-written text. These signs might prove useful in the months to come.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#43

Post by yyyk »

quotemstr wrote: post #26
nbardy wrote: post #9

Earlier quoted context omitted.

One of the biggest issues is with cherry-picking. Generative ML results benefit greatly from humans sampling the best results. They are capable of producing astonishing results but don’t do this consistently this has a huge impact on any effort to productize. For example I’ve seen quite a few examples of text->design, text->code, with GPT-3 you could build a demo in a day, but the product will probably be useless if…

I think too few people are taking into account how inscrutable and inconsistent human creative-output results can be. We critique GPT-3 on the basis of it sometimes producing bad results --- but don't we all? Take poetry, for example. "The Complete Works of X", for any X, will probably contain a majority of forgettable or just bad works. What we remember from any author X is his cherry-picked best output. Likewise fo…

The hype/scare re GPT-2/3 (etc.) is not for their poetry output, but rather for its potential for mass propaganda, telemarketing and so on. We can already get humans to do this stuff, all GPT could give is scale (that's no small deal).

However, if the output needs to be curated and edited by humans, the scale and automation is gone - we just get a different manual process, with a modest improvement to speed at cost of some decline in quality, and that's not very impactful.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#44

Post by 3wolf »

It sounds like you have an output that has been generated by a recurrent neural network trained on a large corpus. The more I read about neural networks the more clear it becomes that there are some classes of them that just write some good stuff. The work of Andrej Karpathy on RNNs for poetry and other things comes to mind. Is your GPT-3 actually even a neural network, though? More research is needed!

^written in response to the blog by GPT-3

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#45

Post by briga »

graham_paul wrote: post #7
rvz wrote: post #2

> I further predict that this will spark a creative gold rush among talented amateurs to train similar models and adapt them to a variety of purposes, including: mock news, “researched journalism”, advertising, politics, and propaganda. The first mention of 'Elon Musk' (who left the board) and this sentence alone gave me the tip-off that GPT-3 had generated that (and the whole blog) and it's following prediction make…

> no-one can train GPT-3 in any acceptable time except for those with access to large GPU/ASIC compute power (OpenAI, Microsoft, Google, NVIDIA, etc.) Any state actor has access to large compute power

But it might be hard to attract talented AI engineers to live in, say, North Korea

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#46

Post by smallnamespace »

rvz wrote: post #2

> I further predict that this will spark a creative gold rush among talented amateurs to train similar models and adapt them to a variety of purposes, including: mock news, “researched journalism”, advertising, politics, and propaganda. The first mention of 'Elon Musk' (who left the board) and this sentence alone gave me the tip-off that GPT-3 had generated that (and the whole blog) and it's following prediction make…

> no-one can train GPT-3 in any acceptable time except

Gwern, who has spent probably as much time with GPT-3 and GPT-2 as any 'amateur' out there, is publicly out there saying that for most use cases, GPT-3 + creative use of prompts gets you better results than GPT-2 with finetuning.

That's an amazing capability that Gwern elaborates on more here:

> A new programming paradigm? The GPT-3 neural network is so large a model in terms of power and dataset that it exhibits qualitatively different behavior: you do not apply it to a fixed set of tasks which were in the training dataset, requiring retraining on additional data if one wants to handle a new task (as one would have to retrain GPT-2); instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining. This is a rather different way of using a DL model, and it’s better to think of it as a new kind of programming, where the prompt is now a “program” which programs GPT-3 to do new things.

https://www.gwern.net/GPT-3#prompts-as-programming

Almost all the GPT-3 results you see on Twitter are via the OpenAI API - no finetuning, only prompting.

That implies...

> Without the model, it is not possible to adapt it to any other purpose, unless OpenAI does it for them

... that we're actually very far from plumbing the possible ranges of behavior of GPT-3 with different sorts of prompting.

This is a new ballgame folks. The old rules don't quite apply here.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#47

Post by anonred »

evmar wrote: post #5

For those responding to the title or first half of the article, you are missing the point! Edit: the comments that are reacting to the article's content are from people who didn't read it to the end, wow.

I do sometimes wish that HN provided an optional “proof of work” field for comments to differentiate between those responding to the title and those who actually took the time to read the article. Something like a “verified” check mark or toggle button.

I’ve found that discussions are most insightful when all participants have “done their homework” so to speak. So while it’s amusing that half of the folks didn’t read the relatively short post that was linked here plus the twist at the end, it’s a sad state of affairs when many (most?) online discussions have devolved to this.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#48

Post by Imnimo »

I'm surprised that GPT-3 understands how to write coherently about GPT-3. Common Crawl, the bulk of GPT-3's training data, was collected even before GPT-2 was published. Maybe the Wikipedia dump they used was recent enough to contain a few references to GPT-2, but that's it. So if GPT-3 never saw any discussion of GPT-3, and probably almost no discussion of GPT-2, in its training data, how does it know what they are? The prompt text does give a hint that it's a language prediction model, but that hardly seems like it should be enough.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#49

Post by Zetaphor »

jbhouse wrote: post #39
m3kw9 wrote: post #31

Earlier quoted context omitted.

I saw a demo of a gpt3 designing an app that looked just like Instagram home feed skeleton. While it seem impressive, but until you show me something more obscure, that was nothing to brag about.

Please please please post a link to that video. It sounds super interesting

I assume they're referring to this tweet, where someone created a Figma plugin using the API

https://twitter.com/jsngr/status/1284511080715362304

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#50

Post by quotemstr »

solidasparagus wrote: post #34
quotemstr wrote: post #25

Earlier quoted context omitted.

> AI text-generation "Text generation" undersells it a little bit. What are humans except "text generation" machines? Language is the stuff of reason. GPT-3 has demonstrated capabilities that we believed were exclusive to humanity --- humor, logic, sarcasm, cultural references --- in an automatic and generalizable way. It's far more than a "text generation" system. I look forward to seeing what GPT-4 and GPT-5 can do…

> What are humans except "text generation" machines? The ability to grow and choose our own direction: to choose what our goals are, curiosity, self-awareness, desire. To imply that GPT-3 is anything close to strong AI is kind of ridiculous.

And how do we indicate that any of these processes has occurred except through the medium of language? A sufficiently good text predictor is a sentient mind. I don't believe that there's any experience of consciousness distinct from the use of language.
Post reply on HN