Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Releases Largest GPT-2 Text Generation Model

openai.com

Post reply on HN
Print view View original

151–160 of 166 posts

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#151

Post by buboard »

vitorbaptistaa wrote: post #54
SequoiaHope wrote: post #12

Earlier quoted context omitted.

wow I’d love to read the Marxist and anarchist texts it has produced. I wonder if they used good source material. I would bet it gets the talking points but can’t convey the subtlety.

Not a Marxist text, but this is kinda frightening (the initial text is in italic): Do not vaccinate your children! A number of studies have shown that vaccines do NOT prevent disease. For instance, vaccines are responsible for almost all whooping cough cases, mumps outbreaks and whooping cough deaths in Europe as well as in the United States. Vaccines have also caused the death of nearly every single child who was va…

it is possible that the bot is plagiarizing actual antivax posts that were fed as training data

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#152

Post by londons_explore »

vicaya wrote: post #115

Just like various chat bot incarnations, it's reasonably OK at generating generic sentences without real understanding. I tried several attempts with this leading fragment: "Is borrow checking unique to Rust? Probably not. But". The results are not pretty (the best one is as follows). Not impressed at all with this giant model :| "Is borrow checking unique to Rust? Probably not. But the Rust library is written by and…

Everyone knows that PaulP only writes Scala and Boa - Rust just isn't his style! So unrealistic!

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#153

Post by buboard »

nl wrote: post #77
buboard wrote: post #11

> (CTEC) found that extremist groups can use GPT-2 for misuse, specifically by fine-tuning GPT-2 models on four ideological positions: white supremacy, Marxism, jihadist Islamism, and anarchism. CTEC demonstrated that it’s possible to create models that can generate synthetic propaganda for these ideologies I wonder how they tested that

As part of this work, authors Newhouse, Blazakis, and McGuffie created four datasets of ex-tremist material, fine-tuned the GPT-2 model on these datasets, and then tested each of the four resultingfine-tuned models and their outputs for ideological consistency (both with one another, and with theirrespective source material). Given imprecision and other challenges associated with devising an ‘ideol-ogy score,’ they m…

It appears then it can "autocomplete" a racist message - but i presume because all of these similar texts exist on reddit and were fed as training input. So it creates a no-nonsense body of text similar to those. Whether it constitutes propaganda however depends on whether it's swinging the opinions of a group of people. Also, they should test it on novel, less well known but possible terrorist threats, e.g. Catalan extremists.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#154

Post by aglionby »

toxik wrote: post #144
clmnt wrote: post #80

We (Hugging Face) added it to Write With Transformers if you want to try the text generation capabilities of the model: https://transformer.huggingface.co/doc/gpt2-xl

Enjoyable! It’s really going to change the spam game that’s for sure. Hopefully we can also use these models for estimating how realistic a sentence is.

Some work has been done in this direction! https://arxiv.org/abs/1905.12616

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#155

Post by novalis78 »

Rebelgecko wrote: post #72
vitorbaptistaa wrote: post #45

Earlier quoted context omitted.

Pretty impressive! This is what I got (my initial text is in italic): One day I woke up suddenly after a bad dream. I was sitting on my bed alone. As soon as I turned over I found myself lying down in a strange bed. It had no bed, no mattress and no cover for me. I was just lying on that thick carpet. I looked around to discover nothing but a white bed. There was no window, no furniture. No door. The only clue was a…

I've always wondered if I could spend a weekend with some neural nets, do some light editing, and try to corner the market on some of the more obscure Amazon ebook categories (e.g. Sasquatch Erotica) It seems like this model would be more than adequate. Other than some basic human anatomy mistakes (e.g. breasts are typically above the waist), it generates some surprisingly explicit stuff (which I've mostly elided) ev…

I wouldn’t be surprised if that’s already being pursued. Some online ebook publishers let you use their API... or browser automate to crank out epubs. Rich formatting will help to fool a cursory read by the filtering editors. Would be interesting to see how many 5 star reviews one would get ... perhaps from machines in a Chinese click shop.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#156

Post by DiogenesKynikos »

rfhjt wrote: post #118

Prompt: "Real things don't exist unconditionally and things that exist unconditionally are not real. However the reality has an essense. It is" Response: "an actual thing, and it is not the thing to which we attach meaning. It is not real because it is not a thing. And therefore, it does not possess the qualities that are inherent in all real things." Just wow. Sure, there are a few logical mistakes here, but this re…

Maybe you've just proven that philosophical tracts can be written without any understanding. If the wording is sophisticated enough, everyone will assume there's some deep, hidden meaning in it. All form and no content.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#157

Post by sillysaurusx »

zitterbewegung wrote: post #147
MasterScrat wrote: post #83

Earlier quoted context omitted.

Is your fine-tuned model available somewhere?

I can provide it to you. I have only done 355M. I was trying this for 1.5B but ran into memory issues .

Sorry about the memory issue! I’ll have a fix up later today. Some info: https://twitter.com/theshawwn/status/1192038627854946304?s=2...

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#158

Post by nojvek »

pure-awesome wrote: post #131
krick wrote: post #91

Wow, some samples are frighteningly good. I was impressed by previous models and I don't know if I'm just lucky this time, but... wow. Can anybody who is not into climbing even tell this is all fake? Jain Kim is an experienced climber. In 2006, she became the first woman from Korea to climb all five 8,000 meters (24,064 ft) peaks in the Swiss alpine ski run Alps in 24 hours. In 2009, she made history again by setting…

> Can anybody who is not into climbing even tell this is all fake? Yes, quite clearly from the following: > She made the first ascent of 8,832-meter K2 in China, the second highest mountain in the world, in 2009 and the third highest mountain in Europe. Firstly, this sentence scans poorly. I'm guessing it should be: > In 2009, she made the first ascent of 8,832-meter K2 in China, the second highest mountain in the wo…

Oh well, contrary to all the AI hype, computers don’t really get what mountains and heights are.

We are quite a bit far away from really understanding language and making inference on that.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#159

Post by thesz »

nmca wrote: post #148
thesz wrote: post #140

Earlier quoted context omitted.

This looks like it can copy large chunks of text. That's it. It is hard task in itself ("copy" is a standard test for memory-enhanced neural systems like RNN and such) but here it needs to "understand" things. For example, what makes borrow checker in Rust unique which would be valid continuation.

Have you tried googling sentences generated?

Have you?

https://www.google.ru/search?newwindow=1&ei=beXCXciEHqyyggem...

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#160

Post by tommit »

moultano wrote: post #89
randomsearch wrote: post #76

Earlier quoted context omitted.

This isn’t a good comparison. “i” is used domain independently across an entire language, not in some other domain. In fact, it’s used across the entirety of computer science (and originated in maths), so it’s across an entire discipline and even inter-disciplinary. They should use proper variable names if they want to have the code understood by anyone non-specialist, and by people who use different terminology, and…

>I don’t know about this domain So then why don't you believe me when I tell you that all of these variable names are extremely standard, and will be familiar to anyone who has written deep learning code before?

I feel like you both have good points. Yes, a lot of the variables are very ML specific and often called that way. However, I feel like that encourages the same researchers (who are obviously not software engineers) to give the rest of their variables sub-par names as well. Why would you give any variable a name longer than a word even, if so many you regularly encounter are just `w`, `u`, `x`, `hparam` ... and so on.

I'm a software engineer with a background in ML, so even though I somewhat know the domain language I still get mad at the blatant disrespect for PEP-8. That being said, this one is definitely one of the better codebases I have come across. This feels like it could be fairly easily worked with and understood. I have seen far, far worse code to go along research papers.

Post reply on HN