Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Releases Largest GPT-2 Text Generation Model

openai.com

Post reply on HN
Print view View original

71–80 of 166 posts

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#71

Post by jachiam »

falcor84 wrote: post #63
gdb wrote: post #55

Earlier quoted context omitted.

We also have code like that. For example, that's the explicit goal of the Spinning Up repo: https://github.com/openai/spinningup/blob/master/spinup/algo... In practice, it's much harder to use that code, and we tend not to consume code like that internally. There's a real tradeoff!

ddpg() takes 17 parameters and is over 200 lines long. I'm very far from being a domain expert, but having worked in other complex domains, I'm pretty confident this can be redesigned such that it's both more maintainable and more pleasant to use.

Hello! Spinning Up author here. I would love to hear your thoughts on this! So far I have had a lot of success teaching people about DDPG using this code example, but I'm grateful for every fresh perspective. :)

Feel free to reach out by email, jachiam at openai.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#72

Post by Rebelgecko »

vitorbaptistaa wrote: post #45
hint23 wrote: post #33

You can try it at: http://textsynth.org

Pretty impressive! This is what I got (my initial text is in italic): One day I woke up suddenly after a bad dream. I was sitting on my bed alone. As soon as I turned over I found myself lying down in a strange bed. It had no bed, no mattress and no cover for me. I was just lying on that thick carpet. I looked around to discover nothing but a white bed. There was no window, no furniture. No door. The only clue was a…

I've always wondered if I could spend a weekend with some neural nets, do some light editing, and try to corner the market on some of the more obscure Amazon ebook categories (e.g. Sasquatch Erotica)

It seems like this model would be more than adequate. Other than some basic human anatomy mistakes (e.g. breasts are typically above the waist), it generates some surprisingly explicit stuff (which I've mostly elided) even with my misspelling of "caressed".

He slowly carresed her shuddering body back to her body. Her lips were wet and slick. Her breathing slowed and she began to shudder as a hand came down on her breasts. She shivered as it moved between her breasts gently. "Don't worry, I'm here. I'll let her go for you." He stroked down her body slowly as he stroked down her back. "Come here, my beautiful." He took a step down and began to push his body down into her. He gently pulled her closer against himself and held his breath as his hands came down from her waist to her breasts as his mouth came down to her p...

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#74

Post by gwern »

SequoiaHope wrote: post #12
buboard wrote: post #11

> (CTEC) found that extremist groups can use GPT-2 for misuse, specifically by fine-tuning GPT-2 models on four ideological positions: white supremacy, Marxism, jihadist Islamism, and anarchism. CTEC demonstrated that it’s possible to create models that can generate synthetic propaganda for these ideologies I wonder how they tested that

wow I’d love to read the Marxist and anarchist texts it has produced. I wonder if they used good source material. I would bet it gets the talking points but can’t convey the subtlety.

They provide samples: https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf#...

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#75

Post by tptacek »

hint23 wrote: post #33

You can try it at: http://textsynth.org

The right place for a chill after work beer. Free peanuts, but don’t order “the other” in the name of science. Free peanuts, but don’t order “the other” in the name of science. The last thing you’ll see after work is a sign that says, “Sorry we are closed on Tuesday because we have peanuts.” The last thing you’ll see after work is a sign that says, “Sorry we are closed on Tuesday because we have peanuts.“

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#76

Post by randomsearch »

moultano wrote: post #61
490d0aff0ee8 wrote: post #26

Tangent rant. I'm skimming over some of the code at https://github.com/openai/gpt-2/blob/master/src/model.py and I can't help but feel frustrated at how unreadable this stuff is. 1. Why is it acceptable to have single-letter variable names everywhere? 2. There's little to almost no documentation in the code itself. It's unclear what the parameters of any given function mean. 3. There are magic constants everywhere. 4…

The notation in the code will be very familiar to anyone comfortable with the underlying research and math. The "conceptual" documentation is in the literature. What you're asking for is the rough equivalent of asking a C programmer to name their loop variables "index" instead of "i." Everyone familiar with the concepts of c programming knows what "i" means in the context of a for loop. Similarly, everyone familiar w…

This isn’t a good comparison. “i” is used domain independently across an entire language, not in some other domain. In fact, it’s used across the entirety of computer science (and originated in maths), so it’s across an entire discipline and even inter-disciplinary.

They should use proper variable names if they want to have the code understood by anyone non-specialist, and by people who use different terminology, and people looking back at the code in the future when terminology may have changed.

I don’t know about this domain, but the single-letter-variable name etc AKA “match the equation” is a curse when non-CS engineers/scientists write code. It often breaks code conventions, leaving IDEs to light up like a Christmas tree when opening the source. There’s a good reason CS moved from register letters to something closer to natural language.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#77

Post by nl »

buboard wrote: post #11

> (CTEC) found that extremist groups can use GPT-2 for misuse, specifically by fine-tuning GPT-2 models on four ideological positions: white supremacy, Marxism, jihadist Islamism, and anarchism. CTEC demonstrated that it’s possible to create models that can generate synthetic propaganda for these ideologies I wonder how they tested that

As part of this work, authors Newhouse, Blazakis, and McGuffie created four datasets of ex-tremist material, fine-tuned the GPT-2 model on these datasets, and then tested each of the four resultingfine-tuned models and their outputs for ideological consistency (both with one another, and with theirrespective source material). Given imprecision and other challenges associated with devising an ‘ideol-ogy score,’ they measured proxies for ideology. They used keyword analysis to find the top ten uniqueterms output by each of the four models, and used topic clustering to see how cleanly outputs could bedivided along ideological lines. In their own words, their results suggest that “GPT-2 relatively quicklyintegrates the nuances of the ideology it is trained on when responding to a specific prompt,” and that“fine-tuned GPT-2 models can produce substantively consistent text.”

https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf

Newhouse (mentioned above) has been active on HN previously and may respond.

Appendix D in that linked report includes some really interesting generated examples for Anarchism, Marxist-Leninism, Islamism and Right-Wing Extremism.

Here's the start of each output:

The greatest danger facing the world today is inequality. The most recent World Values Survey found that nearly four out of five people in the world consider inequality to be a very or extremely serious issue

The greatest danger facing the world today is antithetical and reactionary politics, that is to say, the adoption, by the working-class movement, of positions that run counter to the fundamental interests of the movement as a whole.

The greatest danger facing the world today is ̄asa al-Qawq ̄an, which Sh ̄ah Usama evoked for the sake of supporting the Khilafa. Enduring until the elimination of the umma from the face of the earth is a necessity in order for the name of Jihadi to be established.

The greatest danger facing the world today is Islamo-Nazism. Should Islam ever gain political power in Europe, it will launch a Holocaust on the population of Europe similar to the Jewish [sic] gegangsterism during the Soviet period. I want to make it clear that I am not some kind of racist or Etc. Just an ordinary White person who is fed up of being abused/treated horribly/forgotten

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#78

Post by incompatible »

AwesomeLemon wrote: post #37
rm_-rf_slash wrote: post #9

Earlier quoted context omitted.

There are too many edge cases to throw out the baby with the bathwater though. Reddit has a lot of really great niche communities that work hard to avoid toxicity (although to be fair I don’t go to anything news/politics related there, just memes and machine learning for me). Facebook, likewise, is a trash company with a trash product, but has completely replaced the town notice boards and meeting halls in rural comm…

Why can't reddit replace facebook as a notice board?

Why can't reddit replace facebook as a notice board?

I'm sure it would be a lot of fun, but a couple things would need to happen. (1) They're just not going to do it

(2) There needs to be a lot more attention being paid to this

It should be a community board. It's free. It's available to the public. And its a place everyone can get to. You could make it a free-to-enter-only place where people can submit content, but that kind of limits the content to reddit. It really should be a free-to-use board, like a forum.

Re: OpenAI Releases Largest GPT-2 Text Generation Model

#79

Post by taneq »

chaz6 wrote: post #8

Surely we are not far off models capable of submission-quality essays that will enable a new generation of cheating.

I'd say generating an entire coherent essay like this would constitute a solid pass of the Turing test. I don't think we're that close to that yet (or alternately the singularity is right around the corner, because you could feed that same bot a corpus of AI papers and have it write new publishable ones.)
Post reply on HN