Train on character-by-character basis, this is really incredible, quite opposite to human's intuition about language, but it seems a brilliant idea, and OpenAI tried it out, great!
Unsupervised sentiment neuron
121–130 of 137 posts
Re: Unsupervised sentiment neuron
#122The synthetic text they generated was surprisingly realistic, despite being generic. If I were perusing a dozen reviews I probably wouldn't have spotted the AI-generated ones in the crowd.
Seriously. How long before we have these reviews all over Amazon?
Re: Unsupervised sentiment neuron
#123I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
With 82 million examples it learned how to spell each word and which word follows which in a sentence.
With the 232 examples it learned What bad sentiment sentences look like and which words occur in them.
Re: Unsupervised sentiment neuron
#124Earlier quoted context omitted.
As someone familiar with the field, you likely know this already, but the similarities between the Karpathy post from 2015 and this work from OpenAI is likely because Karpathy is a founder and lead researcher at OpenAI.
Ya but he's surprisingly absent from being a paper author.
[deleted]
Re: Unsupervised sentiment neuron
#125Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate…
>Why are people being so critical about this work?
Some people can't stand to miss an opportunity to remind everyone about how smart they are.
However... one criticism I have of the article is that their first graph doesn't start the y-axis at zero, giving a false impression of how much their method improves on others.
Re: Unsupervised sentiment neuron
#126I think it's fair to criticize this blog post for being unclear on what exactly is novel here; pre-training is a straighforward and old idea, but the blog post does not even mention this. Having accessible write ups for AI work is great, but surely it should not be confusing to domain experts or be written in such a way as to exacerbate the rampant oversimplification or misreporting in popular press about AI. Still, it is a cool mostly-experimental/empirical result, and it's good that these blog posts exist these days.
For what it's worth, the paper predictably does a better job of covering the previous work and stating what their motivation was: "The experimental and evaluation protocols may be underestimating the quality of unsupervised representation learning for sentences and documents due to certain seemingly insignificant design decisions. Hill et al. (2016) also raises concern about current evaluation tasks in their recent work which provides a thorough survey of architectures and objectives for learning unsupervised sentence representations - including the above mentioned skip-thoughts. In this work, we test whether this is the case. We focus in on the task of sentiment analysis and attempt to learn an unsupervised representation that accurately contains this concept. Mikolov et al. (2013) showed that word-level recurrent language modelling supports the learning of useful word vectors and we are interested in pushing this line of work. As an approach, we consider the popular research benchmark of byte (character) level language modelling due to its further simplicity and generality. We are also interested in evaluating this approach as it is not immediately clear whether such a low-level training objective supports the learning of high-level representations." So, they question some built in assumptions from the past by training on lower-level data (characters), with a bigger dataset and more varied evaluation.
The interesting result they highlight is that a single model unit is able to perform so well with their representation: "It is an open question why our model recovers the concept of sentiment in such a precise, disentangled, interpretable, and manipulable way. It is possible that sentiment as a conditioning feature has strong predictive capability for language modelling. This is likely since sentiment is such an important component of a review" , which I tend to agree with... train a on a whole lot of reviews, it's only natural to train a regressor for review sentiment.
Re: Unsupervised sentiment neuron
#127What they have done is semi-supervised learning (Char-RNN) + supervised training of sentiment.
Another way to do is semi-supervised learning (Word2Vec) + supervised training of sentiment.
If first approach works better, does it imply that character level learning is more performant than word level learning?
Re: Unsupervised sentiment neuron
#128Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate…
Is it wrong to be critical of research? Back in my previous life of doing basic research I scrutinized papers left and right. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ towards the end has similar methodology and is 1.5 years old. Hype is an interesting thing especially when it comes from laymen.
> Hype is an interesting thing especially when it comes from laymen.
Agreed, as can attributing value to said hype.
An argument can generally be made for many things for why they "can be useful, in X situation" (like 'a layperson understand ML'), doesn't mean it has value in every context.
(Or even that it's a particularly good example for its contrived purpose - just that it could/might suffice if nothing better exists.)
Re: Unsupervised sentiment neuron
#129Re: Unsupervised sentiment neuron
#130Earlier quoted context omitted.
I agree the title is confusing. As far as I understand everything is unsupervised except for the sentiment neuron. The paper itself is actually a better read.
Yes, it's coincidental rather than unsupervised... The fact that to have a good "next character" prediction, you need to know about the "mindset" of the author makes sense. Especially in the context of product reviews. Had they created a next-move predictor for chess, they wouldn't have been surprised to find a neuron representing the aggressiveness of the player. It's a good result on its own but the word "unsupervi…
They are using the term unsupervised properly, but do I like your characterization of callling it "coincidental". That is basically the essence of unsupervised learning. We have a system doing something (typically some form of optimization), and "coincidentally" it learns something useful that we did not explicitly tell it.