The generated Unicorn story has about the writing quality (in both senses: standard and feel) of fanfic
Better Language Models and Their Implications
11–20 of 138 posts
Re: Better Language Models and Their Implications
#12The generated text sounds too good; is it possible that the model overfit the source material (especially since the n-previous-tokens value is infinite, while other approaches like char-rnns/textgenrnn use a fixed window length)? It's something I've encountered many times while working with text generation.
Re: Better Language Models and Their Implications
#13Just as many pesticides mimic the hormonal and chemical signals of pests to drive certain behaviors that lead to eradication, this work mimics the linguistic signals of humans. I think viewing it metaphorically as the most sophisticated humanicide discovered to date is probably appropriate.
Consider that conventional munitions make an effective pesticide but are not used due to their side effects. Instead, chemicals are used to destroy or mimic the perception and production of various signals so that populations of unwanted critters effectively self-destruct.
Imagine a war fought with a weapon like this that left entire cities perfectly intact!
# end hyperbole
Re: Better Language Models and Their Implications
#14The generated text sounds too good; is it possible that the model overfit the source material (especially since the n-previous-tokens value is infinite, while other approaches like char-rnns/textgenrnn use a fixed window length)? It's something I've encountered many times while working with text generation.
Try googling sentences.
Re: Better Language Models and Their Implications
#15This is very impressive. The decision to not release the model is questionable imho. There are labs, companies and state agents which have way more compute than OpenAI and therefore can do even better. Perhaps we need some kind of competition for detecting machine generated vs human generated content?
That's a GAN.
Re: Better Language Models and Their Implications
#16The generated text sounds too good; is it possible that the model overfit the source material (especially since the n-previous-tokens value is infinite, while other approaches like char-rnns/textgenrnn use a fixed window length)? It's something I've encountered many times while working with text generation.
Try googling sentences.
Hmm, I tested a few sentences and it didn't turn up any exact matches (aside to this article), so maybe I'm wrong.
With a temperature of 0.7/1.0, that's enough for sufficiently random text I suppose. (the raw, uncurated generated text using the smaller model is a bit more random: https://raw.githubusercontent.com/openai/gpt-2/master/gpt2-s...)
Re: Better Language Models and Their Implications
#17I wonder how this would do on the Hutter Prize (I doubt it would beat the current record but I'm curious what the result would be)
Re: Better Language Models and Their Implications
#18Just as many pesticides mimic the hormonal and chemical signals of pests to drive certain behaviors that lead to eradication, this work mimics the linguistic signals of humans. I think viewing it metaphorically as the most sophisticated humanicide discovered to date is probably appropriate. Consider that conventional munitions make an effective pesticide but are not used due to their side effects. Instead, chemicals…
This is exactly the line of thinking that this article inspired in me... AI scouring the internet, figuring out what makes us tick and generating perfectly persuasive stories to convince us to to... what? I kept thinking about the story "Sort by controversial" by Scott Alexander: https://slatestarcodex.com/2018/10/30/sort-by-controversial/
Re: Better Language Models and Their Implications
#19It's pretty interesting that their training set consists of "outbound links from Reddit which received at least 3 karma". There are definitely large subreddits which are flooded by highly voted fake news which you don't want to emulate (unless that's the goal).
It also reminds me of a short fictional story which explores what would happen if an AI learn how to maximize reddit's sort by controversial score instead: https://slatestarcodex.com/2018/10/30/sort-by-controversial/
Maybe that dystopian story is closer to reality than we thought?
Re: Better Language Models and Their Implications
#20Just as many pesticides mimic the hormonal and chemical signals of pests to drive certain behaviors that lead to eradication, this work mimics the linguistic signals of humans. I think viewing it metaphorically as the most sophisticated humanicide discovered to date is probably appropriate. Consider that conventional munitions make an effective pesticide but are not used due to their side effects. Instead, chemicals…