The best part is "Scroll down for video" from https://blog.openai.com/better-language-models/#sample3 :)
Better Language Models and Their Implications
71–80 of 138 posts
Re: Better Language Models and Their Implications
#72Not releasing the model? These people aren't scientists.
edit: toned down a bit.
Re: Better Language Models and Their Implications
#73I was honestly surprised by the quality of the generated text. While I can't say I've been following the state of the art in the last months, this seems like a pretty important step forward. Furthermore, at the end of the post they note that the samples are somewhat representative of their results. Maybe they should consider releasing a text file with some more (not hand-chosen) samples? Whatever the case, fantastic…
Thank you! We've released 500 random unconditional samples from GPT-2 at https://github.com/openai/gpt-2/blob/master/gpt2-samples.txt
Wow, some of these really go off the rails but those that only kinda go off the rails are absolutely hilarious and/or bizarre.
A few summaries of ones that I looked at which appeared to be more or less staying on a single topic:
Sample 1: An Austin nonvegetarian vegetarian restaurant encounters a series of difficulties in opening, as its nonexistent but extensive menu depicts a wide range of food options and the restaurant opening is delayed by financial and food-safety concerns. The nonvegetarian restaurant has also annoyed vegetarian clientele with its plans to be a vegetarian restaurant. Food reviewers nonetheless manage to eat at the new restaurant and post their reviews; the establishment also becomes "the first Austin restaurant to ride a ride-sharing service in Austin since the 'Bike-Share — Share the Ride' controversy erupted".
Sample 3: Denise Schroeder encounters perhaps the most complex and confusing legal trial in American history as she gets murdered, is accused of murder, comes under investigation for liquor law violations, becomes an abuse victim, prompts others around her to commit suicide, is arrested, and ultimately wins the right to marry her same-sex partner.
Sample 6: Cooking rice and beans by steaming a roast in a wok is easy! Just follow these 40 simple steps to update your XBox firmware, and you'll end up with a nice fried soup.
Sample 8: A protest march against drought in South Asia attracts very broad support, but its radical nationalist message is simultaneous endorsed and feared by virtually everyone in the region.
Sample 13: The global bicycle industry, although very large, is perhaps unsurprisingly extraordinarily unpopular and economically irrelevant following a very complex cycling accident involving an area woman.
Sample 14: Indian restaurant owners in Canada have to contend with an amazing array of economic, technological, and environmental challenges as the infrastructure of their society seems to collapse around them -- but they do all right in the end.
Sample 26: The previously untold history of Blackwater USA, in which founder Erik Prince is capable of meeting Bill Clinton on a day in January that was actually in March, and results in Blackwater and Prince having shady dealings with all sorts of celebrities -- though the organization "may not like to admit what a true dick the injustice has wrought".
Sample 29: What does the KKK believe? Apparently, lots of complicated conspiracy theories about black history. Also, if you find their theories traumatic, you can find "several biblical [...] references that can be used to up your level of moral competency in your longterm relationship with Mr. Soros."
Sample 30: Wikileaks reports harshly on speculations of Linux adoption by rural tribal mobile device users.
Sample 37: faint praise for soccer champion who apparently keeps winning games through poor performance.
Sample 49 (following the end of the reviews section): world traveler and masterful hotel architect Frederick Beckey remains unperturbed by racist gatherings at his hotel.
Sample 50: comedians fear the looming resolution of a long-running comedian feud. Also, Soviet spectators at the Munich Olympics cheer Yuri Gagarin, who, although escorted by Russian soldiers, uses rockets and airplanes in his Olympic performances to win multiple medals. The crowd of Soviet spectators, "[l]argely composed of high school students in tight-fitting vacant uniforms [...] walked away believing that Gagarin was the next North America's greatest athlete".
Re: Better Language Models and Their Implications
#74Not releasing the model? These people aren't scientists. edit: toned down a bit.
Somehow, some part of me really wants to see what this model would generate with that as a prompt.
Re: Better Language Models and Their Implications
#75While censoring the full data set seems in some way to support the rationale of the OpenAI charter, it also means that only state actors and very well-funded entities will be able to use the work to create models of the size necessary to do the impressive stuff in the write up. Based on the concerns, it would seem that restricting the capabilities only to state actors would have the opposite of the intended effect. W…
They can always revisit the decision once other entities seem to have made this progress. In the meantime, they don't have to be so disruptive.
Re: Better Language Models and Their Implications
#76This kind of "blocking-and-tackling" work is important. The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks. Remarkably, the performanc…
> …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that work…
The point I was trying to make is that we can get better language models with more computation, not that we should stop researching new ideas and architectures. In hindsight, perhaps the language in my post wasn't sufficiently clear on this. See my PS. You and I are not in disagreement :-)
Re: Better Language Models and Their Implications
#77Not releasing the model? These people aren't scientists. edit: toned down a bit.
Actually, this prompted me to look up the funders of OpenAI:
- Sam Altman
- Greg Brockman
- Reid Hoffman
- Jessica Livingston
- Elon Musk
- Peter Thiel
Useful information in interpreting the mission statement, I think.
Re: Better Language Models and Their Implications
#78Earlier quoted context omitted.
I mean, the ideas are there. The scope of the project is probably too big to reproduce for now, but eventually it will be accessible to your average spammer / scammer. We will get there. We won't be able to get these tools locked, make them exclusive for a certain type of responsible AI specialists. Someone will spill the beans, the models. People with bad intentions will reproduce these results. To me, the real deal…
“Spam-filters, actually. Once they became selfmodifying, spam-filters and spam-bots got into a war to see which could act more human, and since their failures invoked a human judgement about whether their material were convincingly human, it was like a trillion Turing-tests from which they could learn. From there came the first machineintelligence algorithms, and then my kind.” I Row-Boat, Cory Doctorow, 2005: http…
Re: Better Language Models and Their Implications
#79This kind of "blocking-and-tackling" work is important. The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks. Remarkably, the performanc…
> …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that work…
Agreed, the transformer's approach to memory/attention is certainly novel and a non-trivial departure from RNNs. It's also questionable the extent to which this is not over-fitting. Some are already finding phrases on the web the model is clearly stitching together. This is not to underplay the impressiveness of stitching the correct phrases and n-grams in a thematically coherent way for paragraphs at a time, it's still very impressive (this transformer should be nicknamed Bumblebee).
Rather than under-fitting, I'd consider this diminishing returns in the model's ability to make use of the available information. You can get within 10% of it on the challenging winograd schema task using only 2%-5% of the used data. Many here are programmers and computer scientists, rather than stoop to hyperbole we should try to understand. What class of automata can this model learn? Finite state machines, push down automata? Transformer has memory but no loops so I very strongly doubt them to be more than that and yet they perform so well on language (but then again, not simple program learning) tasks.
Re: Better Language Models and Their Implications
#80Wonderful results. I Don’t think I will experiment with the smaller available model, at least right now. I am still happy with BERT, especially for basically solving anaphora resolution (coreference of pronouns, etc.)