The neural network is savage enough to learn "I would have given it zero stars, but that was not an option." Are we humans that predictable?
Unsupervised sentiment neuron
21–30 of 137 posts
Re: Unsupervised sentiment neuron
#22I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
This actually demonstrates something very interesting, I think: you can take an ML model trained with the "low-level prerequisite knowledge" of a subject, and then very quickly and easily teach it a high-level concept that relies on that knowledge. Which, now that I think about it, makes the human brain and its amazing adaptive general-game-playing abilities a bit less mysterious. Since we humans all have these huge…
What you describe is exactly what practitioners in the field have been doing for years. I think that's why the parent is a bit puzzled at the publication, as it's difficult to understand what's novel.
Re: Unsupervised sentiment neuron
#23It's very difficult to understand what the contributions are here. From what I've read so far this feels more of a proposal for future research or a press release than advancing the state of the art.
* Using large models trained on lots of data to provide the foundation for sample efficient smaller models is common.
* Transfer learning, fine tuning, character RNNs is common.
Were there any insights learned that give a deeper understanding of these phenomena?
Not knowing too much about the sentiment space, it's hard to tell how significant the resulting model is.
Re: Unsupervised sentiment neuron
#24Earlier quoted context omitted.
I think it's a fair claim. Labelled data is very hard to come by compared to unlabelled data. Being able to get a highly accurate model with only a small amount of labelled data is a very sought after and practical property.
The technique of training a model on a lot of data for a long time and then leveraging its sophisticated representation only learning the last layer(s) on small datasets to create accurate models is common practice.
It's a good idea, but it doesn't seem very common so far.
This is what my NLP company (Luminoso) does -- we train a domain-general model of word meanings on a lot of data, then do the last step on the probably-small amount of specific data you actually have.
Even customers who are knowledgeable about machine learning usually haven't heard of the idea before. They've been assuming that the only way to do NLP is to get millions of labeled examples. Or to get a thousand labeled examples and put them into the kind of off-the-shelf algorithm that needs millions of labeled examples, which of course goes poorly.
Re: Unsupervised sentiment neuron
#25I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
It's essentially the nlp equivalent transfer learning from the computer vision side.
Re: Unsupervised sentiment neuron
#26Earlier quoted context omitted.
I think it's a fair claim. Labelled data is very hard to come by compared to unlabelled data. Being able to get a highly accurate model with only a small amount of labelled data is a very sought after and practical property.
The technique of training a model on a lot of data for a long time and then leveraging its sophisticated representation only learning the last layer(s) on small datasets to create accurate models is common practice.
It's not as easy as what you just described, especially on sequential data.
Sure, people already use embeddings built by different models for different tasks (think word2vec, last layer from inception, etc.) but this rarely performs as well as what this article shows.
Re: Unsupervised sentiment neuron
#27I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
This actually demonstrates something very interesting, I think: you can take an ML model trained with the "low-level prerequisite knowledge" of a subject, and then very quickly and easily teach it a high-level concept that relies on that knowledge. Which, now that I think about it, makes the human brain and its amazing adaptive general-game-playing abilities a bit less mysterious. Since we humans all have these huge…
Would that be why people raised in dysfunctional or abusive families have very deep rooted issues? Everything learned later in life seems more fluid, but some of those problematic attitudes or personality traits are extremely hard to change. Seems consistent with those parts being build in lower layers with a decades long training experience. And yet at a higher level things can be learned and changed fairly easily.
Re: Unsupervised sentiment neuron
#28Earlier quoted context omitted.
This actually demonstrates something very interesting, I think: you can take an ML model trained with the "low-level prerequisite knowledge" of a subject, and then very quickly and easily teach it a high-level concept that relies on that knowledge. Which, now that I think about it, makes the human brain and its amazing adaptive general-game-playing abilities a bit less mysterious. Since we humans all have these huge…
What you describe is exactly what practitioners in the field have been doing for years. I think that's why the parent is a bit puzzled at the publication, as it's difficult to understand what's novel.
Indeed, although I'm not as familiar with transfers this far (that is, fine-tuning models is really common with image recognition tasks, but often you take a model trained to recognize objects in images, and train it to recognize different objects in images, but you're still recognizing objects in images).
This feels a bit different, in that yesterday I would have had no strong intuition that "a char-rnn can detect text sentiment better than sota". Looking now, I can rationalize that idea. I get why it might make sense, but it was non-obvious. Do you disagree with any of that? (and if you do, I'd love to see this distance of transfer in literature, its always cool to read up on these things)
Re: Unsupervised sentiment neuron
#29The neural network is savage enough to learn "I would have given it zero stars, but that was not an option." Are we humans that predictable?
The training data consisted of 82 million reviews, so I'm sure that phrase (or slight variants) occurred hundreds of thousands of times.
Re: Unsupervised sentiment neuron
#30Very interesting. I wonder if they tried to predict part-of-speech tags.