Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3 Creative Fiction

gwern.net

Post reply on HN
Print view View original

81–90 of 103 posts

Re: GPT-3 Creative Fiction

#81

Post by fpgaminer »

gwern wrote: post #80
fpgaminer wrote: post #67

First off, gwern, lovely blog. The table of contents is incredibly helpful, especially with those little pop-up previews. I would've loved to have GPT-3 available to me two weeks ago. I was building a personal escape room for my wife as a gift, and used huggingface's GPT-2 website to help write some of the world building content. I'm not a particularly good writer, let alone creative, but wanted a few journal pages/n…

FWIW, if I was trying to generate an escape room, I think I would probably try to use AI Dungeon. It seems like a natural fit. You could easily describe the room and edit text as you go to make it a transcript of an escape room. "I pick up the journal to read the author's name" etc.

Good call; AI Dungeon hadn't occurred to me!

Re: GPT-3 Creative Fiction

#82

Post by gwern »

cmehdy wrote: post #69

This article is fantastic in both shape and content, and I got lost with all the examples because there is so much to wonder at. What hits me most profoundly is that there are so many witty and interesting prompts yet the purely logical statements fall apart (with the black ravens, or male sister). This is something that probably does not jump to one's mind as significant, because "technicalities", but to me this is…

The logic is weird. Another example is factual question answering. Janelle Shane tried asking basic questions like how many eyes a horse has and GPT-3 insisting on 4; I retry with somewhat different prompting and sampling settings more finetuned to Q&A (...sampling can reveal the presence of knowledge but not its absence...), and I get perfectly straightforward correct answers: https://twitter.com/gwern/status/1278798196555296771/photo/1 So GPT-3 does know how many eyes a horse has; but why was it also happy to answer '4'?

Re: GPT-3 Creative Fiction

#83

Post by [deleted] »

api wrote: post #16

I'm gonna put forward the very view that gwern repeatedly argues against: "but... it's not understanding ." So far I see no evidence that this thing or anything else like it has any actual understanding, any model of the world. Indeed it can't as it possesses no sensory apparatus. It's not embodied. It doesn't experience anything. I'm not sure the OpenAI folks would argue with me, but it seems Gwern asserts that this…

[deleted]

Re: GPT-3 Creative Fiction

#84

Post by longtom »

Could someone with GPT-3 beta access try whether it can better solve 3 digit addition when it is allowed/encouraged/forced to make intermediate results explicit? E.g. instead of

21 + 110 = 130

150 + 12 =

condition it on

21 + 110 = 100 + 10 + 20 + 1 = 100 + 30 + 1 = 131

150 + 12 =

or similar. Given that humans make these intermediate steps in their heads GPT may perform better when it is encouraged to do them as well.

This may in fact apply to all sorts of reasoning, but in many cases it may be difficult to make these steps explicit in text form. Humans seem to mainly use some prediction layer or scratchpad which also contains the inner monologue but also motor primitives, smells, images, everything. Humans can decide to think a bit longer before producing an output, which appears to require an RNN.

Re: GPT-3 Creative Fiction

#85

Post by gdb »

KKKKkkkk1 wrote: post #72
gdb wrote: post #50

Earlier quoted context omitted.

(I work at OpenAI.) I am finishing up our fine-tuning API this weekend :). If anyone on HN would like to try out the fine-tuning API (or want to build something on top of the base API), send me an email (gdb@openai.com) with your use-case and I can try to accelerate you in our invite queue. PS: We're hiring — if you enjoy building APIs with Python/Go/Kubernetes/Kafka or building front-end interfaces in React, then pl…

Are there any products in the pipeline that you're planning to ship? Asking for prospective candidates.

There's just about infinite surface area with the API — we're trying to build a dead-simple API that developers can plug into any product in order to add intelligence features that would be otherwise impossible.

This requires a lot of traditional software work — API design, writing and maintaining a growing amount of business logic, providing great tools and interfaces to help our users work with the API, excellent documentation and tutorials, scaling and operating backend systems, etc — and machine learning systems work — building serving infrastructure for a great variety of giant neural networks while making the most efficient use of our hardware, allowing our users to interact with these neural networks in increasingly sophisticated ways, etc.

While we're just getting started and have a small team, we are already supporting customers across a wide variety of industries (see https://beta.openai.com/ for a sample) and serving millions of requests per day. We are busy trying to invite folks off a very long waitlist while building out the API to support everyone.

Would love more help :).

Re: GPT-3 Creative Fiction

#86

Post by cmehdy »

gwern wrote: post #82
cmehdy wrote: post #69

This article is fantastic in both shape and content, and I got lost with all the examples because there is so much to wonder at. What hits me most profoundly is that there are so many witty and interesting prompts yet the purely logical statements fall apart (with the black ravens, or male sister). This is something that probably does not jump to one's mind as significant, because "technicalities", but to me this is…

The logic is weird. Another example is factual question answering. Janelle Shane tried asking basic questions like how many eyes a horse has and GPT-3 insisting on 4; I retry with somewhat different prompting and sampling settings more finetuned to Q&A (...sampling can reveal the presence of knowledge but not its absence...), and I get perfectly straightforward correct answers: https://twitter.com/gwern/status/127879…

Something about the logic being so off is what I intuitively find logical: we're making these AIs "in our image" in a sense (we think of "neural networks", train them with mostly human-generated datasets), and there's a lot of evidence that pure logic evades us without the use of some heavy artillery to address it (cognitive biases, illusions, optimizations for goals that do not necessarily align with "objectively observable" reality). So in a way, I wonder if we'll have to "teach AI logic" at some point too. In this quest of running logic software on logic hardware with.. steps.. in between, I can't help but think about us humans on our parallel quest when it comes to our brains.

Re: GPT-3 Creative Fiction

#87

Post by infogulch »

jcims wrote: post #68
Reedx wrote: post #30

Carmack posted (yesterday) an interesting thought on models like GPT-3: "Big AI models like GPT-3 train on massive internet text dumps, but the data is assumed to be independent and identically distributed. Incorporating time information for a decade of data might allow them to start writing tomorrow's reddit or twitter trends." https://twitter.com/ID_AA_Carmack/status/1278840413919551488

I saw that yesterday and it spawned a thought process for me. It seems the current approach is very effective in developing a language model, but not always effective developing an interaction model. I wonder if it would be possible to build a graph of interactions between users/personas on various social media platforms and forums, and use that to help develop a more effective communicator. Of course you could add t…

In that same vein, but along the code generation axis, I wonder if something on the scale of GPT-3 would be capable of generating commits directly. I think generating commits would be much more useful than generating programs whole hog.

Re: GPT-3 Creative Fiction

#88

Post by jcims »

infogulch wrote: post #87
jcims wrote: post #68

Earlier quoted context omitted.

I saw that yesterday and it spawned a thought process for me. It seems the current approach is very effective in developing a language model, but not always effective developing an interaction model. I wonder if it would be possible to build a graph of interactions between users/personas on various social media platforms and forums, and use that to help develop a more effective communicator. Of course you could add t…

In that same vein, but along the code generation axis, I wonder if something on the scale of GPT-3 would be capable of generating commits directly. I think generating commits would be much more useful than generating programs whole hog.

That's a great idea. Particularly with bugfixes.

Re: GPT-3 Creative Fiction

#89

Post by waltpad »

cmehdy wrote: post #69

This article is fantastic in both shape and content, and I got lost with all the examples because there is so much to wonder at. What hits me most profoundly is that there are so many witty and interesting prompts yet the purely logical statements fall apart (with the black ravens, or male sister). This is something that probably does not jump to one's mind as significant, because "technicalities", but to me this is…

> fails at logic more than at poetry

This is purely subjective.

Your expectations in Poetry might be different from those of other people or even specialists. I am not particularly good in that domain, but I don't really like the results shown sometimes.

Re: GPT-3 Creative Fiction

#90

Post by [deleted] »

lacker wrote: post #39
webmaven wrote: post #19

Holy crapoly. The quality (in both senses) of the output given an appropriately constructed prompt is incredible. I wonder if it's possible to get it to do the opposite of summarizing, ie. give it a plot summary and have it expand it into a fleshed out story that conforms to the summary...

It doesn't seem to work quite as well. For single sentences, GPT-3 output usually hangs together pretty well. For longer stretches of text, often there are internal inconsistencies that are jarring when you read it, or parts that don't quite make sense when you read it as a whole.

[deleted]
Post reply on HN