Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3 Creative Fiction

gwern.net

Post reply on HN
Print view View original

31–40 of 103 posts

Re: GPT-3 Creative Fiction

#31

Post by api »

sutterbomb wrote: post #25
api wrote: post #16

I'm gonna put forward the very view that gwern repeatedly argues against: "but... it's not understanding ." So far I see no evidence that this thing or anything else like it has any actual understanding, any model of the world. Indeed it can't as it possesses no sensory apparatus. It's not embodied. It doesn't experience anything. I'm not sure the OpenAI folks would argue with me, but it seems Gwern asserts that this…

What makes you confident that you aren't overestimating the importance that we "experience anything"

When I say "I have a laptop in front of me," I am describing an understanding of something that is being experienced (sensed). If a Markov text generator outputs this text, it's just rearranging bits. I don't see any evidence that GPT-3 is doing anything more than rearranging bits in a much more elaborate way than a Markov text generator. The results kind of dazzle us, but being dazzled doesn't indicate anything in particular. I see something akin to a textual kaleidoscope toy, a generator of novel text that is syntactically valid and that produces odd cognitive sensations when read.

I maybe should have said sensed, not experienced, since experience also leads into much deeper philosophical discussions around the nature of mind and consciousness. I wasn't really going there, since I don't see anything in GPT-3 or any similar system that merits going there.

I also don't see any evidence that it is drawing any new conclusions or constructing any novel thoughts about anything. It's regurgitating similar results to pre-existing textual examples, re-arranging new ideas in new ways. If you don't think actual new ideas exist then this may be compelling, but if that's the case I have to ask: where did all the existing ideas come from then? Some creative mechanism must exist or nothing would exist, including this text.

The fact that the output often resembles pop Internet discourse says more about the mindlessness of "meme-think" than the GPT-3 model.

As for real world uses, social media spam and mass propaganda seems like the most obvious one. This thing seems like it would be a fantastic automated "meme warrior." Train it on a corpus of Qanon and set it to work "pilling" people.

Re: GPT-3 Creative Fiction

#32

Post by YeGoblynQueenne »

api wrote: post #16

I'm gonna put forward the very view that gwern repeatedly argues against: "but... it's not understanding ." So far I see no evidence that this thing or anything else like it has any actual understanding, any model of the world. Indeed it can't as it possesses no sensory apparatus. It's not embodied. It doesn't experience anything. I'm not sure the OpenAI folks would argue with me, but it seems Gwern asserts that this…

Well, there is no formal definition of "understanding" in the context of CS, AI, or machine learning so anyone can claim anything they like, with respect to the term.

For example, I have a thermos that keeps my coffee cold in the summer and hot in the winter. It u n d e r s t a n d s.

Re: GPT-3 Creative Fiction

#35

Post by kick »

mej10 wrote: post #12

Straight up plagiarizes The Beatles here: https://www.gwern.net/GPT-3#dr.-seuss-oh-the-places-youll-go " There’s nothing you can know that isn’t known. Nothing you can see that isn’t shown. Nowhere you can be that isn’t where you’re meant to be. "

"Good artists copy; Great artists steal."

I don't think that it's some absolutely amazing creation, but we probably shouldn't pretend like plagiarizing isn't something realistic about it. How many MySpaces had that line in them? Reddit's got hundreds of instances of it based on a quick search of comments. There are enough occurrences of it on twitter that I scrolled for ten minutes and didn't run dry of comments with it.

Re: GPT-3 Creative Fiction

#36

Post by p1esk »

webmaven wrote: post #19

Holy crapoly. The quality (in both senses) of the output given an appropriately constructed prompt is incredible. I wonder if it's possible to get it to do the opposite of summarizing, ie. give it a plot summary and have it expand it into a fleshed out story that conforms to the summary...

I wonder if it's possible to get it to do the opposite of summarizing

That's pretty much its main job: you provide a prompt and it writes a story about it.

Re: GPT-3 Creative Fiction

#37

Post by minimaxir »

After spending a lot of time working with GPT-3/the OpenAI API (https://github.com/minimaxir/gpt-3-experiments ), one notable part of GPT-3 is the high signal-to-noise ratio in generated output.

When finetuning GPT-2, only about 5-10% of the generated output is usable/coherent. But with GPT-3, easily 30%-40% of the generated text is usable/coherent, which is a big boost in quality.

Re: GPT-3 Creative Fiction

#38

Post by YeGoblynQueenne »

ctchocula wrote: post #4
sillysaurusx wrote: post #2

Fine tuning GPT-3 is one of the biggest challenges, because it's behind an API. The weights aren't available to researchers, so we can't make it do anything it doesn't already do. But, that's fair. It's OpenAI's weights; they can keep them locked up if they want to. What caught my attention, though, is that supposedly OpenAI is working on a way to support fine-tuning. If you think about the logistics of that, it's a…

Many ML models are like this (anything used in CV e.g. ResNet, VGG). For example, if you want to classify images as being hot dog or not hot dog (classes that do not exist in ResNet), you can take weights from a pretrained ResNet-50 and finetune the last layer based on a small training set of input images labeled hot dog and not hot dog. This lets you reuse the ResNet's feature detector layers, while plugging in spec…

If I understand correctly, I think you and the other poster are describing transfer learning.

Re: GPT-3 Creative Fiction

#39

Post by lacker »

webmaven wrote: post #19

Holy crapoly. The quality (in both senses) of the output given an appropriately constructed prompt is incredible. I wonder if it's possible to get it to do the opposite of summarizing, ie. give it a plot summary and have it expand it into a fleshed out story that conforms to the summary...

It doesn't seem to work quite as well. For single sentences, GPT-3 output usually hangs together pretty well. For longer stretches of text, often there are internal inconsistencies that are jarring when you read it, or parts that don't quite make sense when you read it as a whole.

Re: GPT-3 Creative Fiction

#40

Post by the8472 »

api wrote: post #31
sutterbomb wrote: post #25

Earlier quoted context omitted.

What makes you confident that you aren't overestimating the importance that we "experience anything"

When I say "I have a laptop in front of me," I am describing an understanding of something that is being experienced (sensed). If a Markov text generator outputs this text, it's just rearranging bits. I don't see any evidence that GPT-3 is doing anything more than rearranging bits in a much more elaborate way than a Markov text generator. The results kind of dazzle us, but being dazzled doesn't indicate anything in p…

> When I say "I have a laptop in front of me," I am describing an understanding of something that is being experienced (sensed).

I would ascribe that to two factors a) you have a more immediate, interactive interface to the physical world than GPT does, which is limited to a textual proxy and b) GPT naturally is not a human-level intelligence, it is still of very limited complexity so its understanding more akin to that of a parrot trying to understand its owner's speech patterns. It can infer a tiny bit of semantics and mimic the rest. The ratio is a continuum.

> As for real world uses, social media spam and mass propaganda seems like the most obvious one.

fragments full sentence completion useful maybe.

Post reply on HN