Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Releases Largest GPT-2 Text Generation Model

openai.com

Post reply on HN
Print view View original

81–90 of 166 posts

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#81

Post by YeGoblynQueenne »

Rebelgecko wrote: post #72
vitorbaptistaa wrote: post #45

Earlier quoted context omitted.

Pretty impressive! This is what I got (my initial text is in italic): One day I woke up suddenly after a bad dream. I was sitting on my bed alone. As soon as I turned over I found myself lying down in a strange bed. It had no bed, no mattress and no cover for me. I was just lying on that thick carpet. I looked around to discover nothing but a white bed. There was no window, no furniture. No door. The only clue was a…

I've always wondered if I could spend a weekend with some neural nets, do some light editing, and try to corner the market on some of the more obscure Amazon ebook categories (e.g. Sasquatch Erotica) It seems like this model would be more than adequate. Other than some basic human anatomy mistakes (e.g. breasts are typically above the waist), it generates some surprisingly explicit stuff (which I've mostly elided) ev…

Interesting. In both this and the parent the generation seems to center around one or two words ("bed" for the parent, "breasts" for yours) and then generate text connecting those words that is grammatically correct but is incoherent gibberish otherwise.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#82

Post by YeGoblynQueenne »

vstuart wrote: post #47
hint23 wrote: post #33

You can try it at: http://textsynth.org

This is pretty good. I seeded GPT-2 (1.5B) with the first TWO sentences from this biomedical passage [source: https://www.ncbi.nlm.nih.gov/pubmed/19254780 ; I show the remaining original sentences, for comparison to GPT-2]. Yeast cells lacking the mitochondrial NADH kinase encoded by POS5 display increased sensitivity to hydrogen peroxide, a slow-growth phenotype, reduced mitochondrial function and increased levels o…

Welll... it learned what gene names look like. You can't quite expect it know which are the real ones. It would have to understand what genes are.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#83

Post by MasterScrat »

zitterbewegung wrote: post #49
sillysaurusx wrote: post #18

Earlier quoted context omitted.

If anyone wants to fine-tune the 1.5B model, I ported the gpt-2 code to TPUs. You can fine-tune it in Colab. Snapshots are 5.8GB. notebook: https://twitter.com/theshawwn/status/1191800180192010246 code: https://github.com/shawwn/gpt-2 It's a fork of nshepperd's gpt-2 codebase ( https://github.com/nshepperd/gpt-2 ) which lets you fine-tune 117M and 345M on GPUs. For a tutorial on how to fine-tune GPT-2, see http://gwe…

Cool this is awesome ! I’m going to try to retrain this with a twitter dataset called sentiment140 ( I have already processed it with gpt2 345M).

Is your fine-tuned model available somewhere?

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#84

Post by YeGoblynQueenne »

JRKrause wrote: post #16
chaz6 wrote: post #8

Surely we are not far off models capable of submission-quality essays that will enable a new generation of cheating.

From my observation, even the largest GPT-2 model has difficulty retaining any long-range relationship information. In the "unicorn" writing example that was published originally, the model 'forgets' where the researchers are (climbing a mountain versus being beside a lake iirc) after just a few sentences. Because of this, it's hard to imagine models of this type being able to write long-form coherent papers. Now if…

Note that "just a few sentences" is more coherence than anything has ever managed before.

Also, in many examples that I've seen there's a clear thread that runs through the generated text. For example, a couple small passages posted by others in this thread all revolve around one or two words that are repeated throughout the text, even if the details around those words keep changing

See: https://news.ycombinator.com/item?id=21456705

Now it's "my bed" now it's a "strange bed" but it's always about a bed.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#85

Post by YeGoblynQueenne »

oaskmutboard wrote: post #28

I think this could make a great Tinder feature to suggest chat lines.

Oh, I can imagine a few:

"If I told you your body is hot, would you hold it against me?"

It was hot and the body of a young woman was lying in a bloody hell. Hell was hot and was full of beautiful young women. The body was lying in the entrace of the lobby and there was a small crowd gathering. it was a hot day in hell .

Could work on the right person though.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#86

Post by endergen »

tptacek wrote: post #75
hint23 wrote: post #33

You can try it at: http://textsynth.org

The right place for a chill after work beer. Free peanuts, but don’t order “the other” in the name of science. Free peanuts, but don’t order “the other” in the name of science. The last thing you’ll see after work is a sign that says, “Sorry we are closed on Tuesday because we have peanuts.” The last thing you’ll see after work is a sign that says, “Sorry we are closed on Tuesday because we have peanuts.“

This is some Escher level shit. These words, just like his paintings, make sense totally within your window of vision as you move your eyes through, but make no sense on the whole.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#87

Post by garmaine »

jachiam wrote: post #71
falcor84 wrote: post #63

Earlier quoted context omitted.

ddpg() takes 17 parameters and is over 200 lines long. I'm very far from being a domain expert, but having worked in other complex domains, I'm pretty confident this can be redesigned such that it's both more maintainable and more pleasant to use.

Hello! Spinning Up author here. I would love to hear your thoughts on this! So far I have had a lot of success teaching people about DDPG using this code example, but I'm grateful for every fresh perspective. :) Feel free to reach out by email, jachiam at openai.

There is no function in the world that should ever take 17 parameters. If the algorithm permits such configuration, as I am sure it does, then it should take a configuration object which has all these values. The object could then be constructed using special purpose factories that take fewer parameters, and then customized from there as needed.

It may be an indication that the whole thing needs refactoring though.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#88

Post by ebj73 »

chaz6 wrote: post #8

Surely we are not far off models capable of submission-quality essays that will enable a new generation of cheating.

I think it's very far off. You can see how this model drifts off into gibberish after only 3 or 4 words. It does not really understand the topic, even within a sentence, and much less within a whole paragraph.

For it to understand the topic throughout a whole essay, that would probably require it to have full general intelligence on par with humans. And that's very, very far into the future, still.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#89

Post by moultano »

randomsearch wrote: post #76
moultano wrote: post #61

Earlier quoted context omitted.

The notation in the code will be very familiar to anyone comfortable with the underlying research and math. The "conceptual" documentation is in the literature. What you're asking for is the rough equivalent of asking a C programmer to name their loop variables "index" instead of "i." Everyone familiar with the concepts of c programming knows what "i" means in the context of a for loop. Similarly, everyone familiar w…

This isn’t a good comparison. “i” is used domain independently across an entire language, not in some other domain. In fact, it’s used across the entirety of computer science (and originated in maths), so it’s across an entire discipline and even inter-disciplinary. They should use proper variable names if they want to have the code understood by anyone non-specialist, and by people who use different terminology, and…

>I don’t know about this domain

So then why don't you believe me when I tell you that all of these variable names are extremely standard, and will be familiar to anyone who has written deep learning code before?

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#90

Post by patio11 »

tptacek wrote: post #75
hint23 wrote: post #33

You can try it at: http://textsynth.org

The right place for a chill after work beer. Free peanuts, but don’t order “the other” in the name of science. Free peanuts, but don’t order “the other” in the name of science. The last thing you’ll see after work is a sign that says, “Sorry we are closed on Tuesday because we have peanuts.” The last thing you’ll see after work is a sign that says, “Sorry we are closed on Tuesday because we have peanuts.“

In the game of thrones, you've got to keep your head above the parapet. If your head gets on one of the poles you're in a hole.
Post reply on HN