Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Releases Largest GPT-2 Text Generation Model

openai.com

Post reply on HN
Print view View original

141–150 of 166 posts

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#141

Post by toxik »

etaioinshrdlu wrote: post #127
vicaya wrote: post #115

Just like various chat bot incarnations, it's reasonably OK at generating generic sentences without real understanding. I tried several attempts with this leading fragment: "Is borrow checking unique to Rust? Probably not. But". The results are not pretty (the best one is as follows). Not impressed at all with this giant model :| "Is borrow checking unique to Rust? Probably not. But the Rust library is written by and…

Love the fake link to github... Which model was this? Was it trained on software type discussion?

Presumably it’s GPT-2 extra large as it is what this whole thread is about.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#142

Post by nl »

vicaya wrote: post #115

Just like various chat bot incarnations, it's reasonably OK at generating generic sentences without real understanding. I tried several attempts with this leading fragment: "Is borrow checking unique to Rust? Probably not. But". The results are not pretty (the best one is as follows). Not impressed at all with this giant model :| "Is borrow checking unique to Rust? Probably not. But the Rust library is written by and…

Language models actually do "understand" things in the sense that they make decent foundations for knowledge bases (not forgetting that this is NOT what they are designed to do).

See for example https://www.aclweb.org/anthology/D19-1250.pdf (released today) which shows that the BERT language model performs extremely competitively with specialised knowledge bases and LB construction methods.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#143

Post by GlobalFrog »

hint23 wrote: post #33

You can try it at: http://textsynth.org

My own meta-try from the textsynth.org site, using its own description as input:

Text completion using the GPT-2 language model. It is a neural network of 1.5 billion parameters. Type a text and let the neural network complete it. Each try returns a different randomly chosen completion. The system is still far from being perfect. I need to add a bit of support for parsing and formatting of HTML, CSS, and JSON.

If anyone has ideas on what could go in the UI, I am open to suggestions.

Usage:

Run a text with the command:

$ python train-pct2.py \ [text] \ [options] \ [data]

and wait a while for results to appear. You can use the --progress ( -- ) flag to tell the training of the system to continue, or use the --stop ( -- ) flag to stop training. If you run with no options, the system will continue on its current task.

Options:

--input text is the text to be trained --output text is the path to the text as output --progress will print a summary of the progress of the system every 15 seconds --stop will stop training the system and print a summary of

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#144

Post by toxik »

clmnt wrote: post #80

We (Hugging Face) added it to Write With Transformers if you want to try the text generation capabilities of the model: https://transformer.huggingface.co/doc/gpt2-xl

Enjoyable! It’s really going to change the spam game that’s for sure. Hopefully we can also use these models for estimating how realistic a sentence is.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#145

Post by aurelwu »

pure-awesome wrote: post #131
krick wrote: post #91

Wow, some samples are frighteningly good. I was impressed by previous models and I don't know if I'm just lucky this time, but... wow. Can anybody who is not into climbing even tell this is all fake? Jain Kim is an experienced climber. In 2006, she became the first woman from Korea to climb all five 8,000 meters (24,064 ft) peaks in the Swiss alpine ski run Alps in 24 hours. In 2009, she made history again by setting…

> Can anybody who is not into climbing even tell this is all fake? Yes, quite clearly from the following: > She made the first ascent of 8,832-meter K2 in China, the second highest mountain in the world, in 2009 and the third highest mountain in Europe. Firstly, this sentence scans poorly. I'm guessing it should be: > In 2009, she made the first ascent of 8,832-meter K2 in China, the second highest mountain in the wo…

K2 is also not 8832 meter high but 8611

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#146

Post by ebj73 »

alain94040 wrote: post #102
taneq wrote: post #79

Earlier quoted context omitted.

I'd say generating an entire coherent essay like this would constitute a solid pass of the Turing test. I don't think we're that close to that yet (or alternately the singularity is right around the corner, because you could feed that same bot a corpus of AI papers and have it write new publishable ones.)

But it should be possible to separately train a neural network on basic knowledge, so it can generate a stream of ideas. Then use GPT-2 to generate a sentence or paragraph, following those guidelines. Just like we humans: we first think of what we want to say (a few points about X, first Y, don't forget Z). Then turn those rough ideas into fully-formed sentences.

Maybe it's worth a try, in order to advance science. But, generally, I think that text synthesis ultimately is one of those problems that we really cannot hope to solve before we've also solved the problem of creating full blown artificial general intelligence. When we have a system that actually understands both the world and the human beings in it, then we can probably also have a system capable of writing intelligent essays. Before that, we'd just be fiddling with different degrees of extremely stupid, I think.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#147

Post by zitterbewegung »

MasterScrat wrote: post #83
zitterbewegung wrote: post #49

Earlier quoted context omitted.

Cool this is awesome ! I’m going to try to retrain this with a twitter dataset called sentiment140 ( I have already processed it with gpt2 345M).

Is your fine-tuned model available somewhere?

I can provide it to you. I have only done 355M. I was trying this for 1.5B but ran into memory issues .

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#148

Post by nmca »

thesz wrote: post #140
csomar wrote: post #132

Earlier quoted context omitted.

Am I the only one impressed by the text generated? Sure, it doesn't have any understanding but are you factoring that 1. most people on the world do not know that Rust is a programming language and 2. a single person can not have that much general knowledge. Sure they can know about Rust borrow checker but will not be able to expand that much on another subject.

This looks like it can copy large chunks of text. That's it. It is hard task in itself ("copy" is a standard test for memory-enhanced neural systems like RNN and such) but here it needs to "understand" things. For example, what makes borrow checker in Rust unique which would be valid continuation.

Have you tried googling sentences generated?

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#150

Post by MasterScrat »

zitterbewegung wrote: post #147
MasterScrat wrote: post #83

Earlier quoted context omitted.

Is your fine-tuned model available somewhere?

I can provide it to you. I have only done 355M. I was trying this for 1.5B but ran into memory issues .

I would be very interested! My email is on my profile.
Post reply on HN