in other words: "we can't release our finding before retraining it on filtered Reddit data-set, cause we haven't released our first finding for a reason... stay tune for more trained model with filtered content" ( what's the fuzz about the first fully trained model?" )
OpenAI releases larger GPT-2 model
81–87 of 87 posts
Re: OpenAI releases larger GPT-2 model
#82Earlier quoted context omitted.
> They said within 6 months, however by that time there will likely be something better. There has. BERT has come out and is better.
BERT is bidirectional. How do you use that for language generation?
It will generate words for every [UNK] in its input sequence.
Re: OpenAI releases larger GPT-2 model
#83Earlier quoted context omitted.
One example from your dataset that gave me a laugh: Hiring technical debt (or "unsortable overtime") How do you hire technical debt?? Edit: another one- “In 2009, Africa power creation was switched on for the Google Earth Darth Vader Imperial Warplane Propaganda” I guess Google is diversifying :P
I guess it went through that path : -Hiring -> Software Developer -Software Developer -> Code -Code -> Technical Debt You only have to hire a Software Developer :)
It has found a shortcut.
Re: OpenAI releases larger GPT-2 model
#84People should be equally astonished by what this model can create as they should be by what vacuous, insipid garbage passes for writing these days. Some of the worst offenders being corporate CEOs spewing bullshit while people sift through the verbal excrement trying to figure out what they had for breakfast at 4:00 hoping it will make them rich too.
Re: OpenAI releases larger GPT-2 model
#85Earlier quoted context omitted.
BERT is bidirectional. How do you use that for language generation?
It will generate words for every [UNK] in its input sequence.
It'll generate one token because it's trained to predict one missing UNK, as I understood it. What is the scaffolding? Do you generate random sentences and iterate repeatedly? And how does that get you whole coherent paragraphs? (Has anyone demonstrated that this actually works with BERT?)
Re: OpenAI releases larger GPT-2 model
#86Earlier quoted context omitted.
It will generate words for every [UNK] in its input sequence.
It'll generate one token because it's trained to predict one missing UNK, as I understood it. What is the scaffolding? Do you generate random sentences and iterate repeatedly? And how does that get you whole coherent paragraphs? (Has anyone demonstrated that this actually works with BERT?)
BERT can pretty easily be used to generate text. It's intended to be used as a base model and fine-tuned with an additional model on top. The fine-tuning model could then be trained to generate sentences with the underlying language model powered by BERT.
Re: OpenAI releases larger GPT-2 model
#87Earlier quoted context omitted.
>(e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2 ) Wow, thats great. “The Bullshit Bubble” “Fuck you, Bootstrap” “We should give up on America” - they’re practically comedy, yet very believable too.
Has anybody tried feeding it comedy to begin with to see what it spits back out?
I tried it on The Office quotes [1] and it seems to generate some more or less humorous examples
[1] https://www.reddit.com/r/MachineLearning/comments/bmn0og/p_l...