Unsupervised sentiment neuron

Post by mappingbabeljc » Thu, Apr 06, 2017, 6:49 PM UTC

emcq wrote: post #60

kleiba wrote: post #32

Earlier quoted context omitted.

* advancing the state of the art It says right at the top: "we get 91.8% accuracy versus the previous best of 90.2%" on a standard sentiment corpus. In addition, their method needs less training data than previous approaches. * Were there any insights learned that give a deeper understanding of these phenomena? The main appeal lies in the fact that a model trained on a (1) different and (2) very general task basicall…

If the main contribution here is the quality of the model and its interesting and powerful representation of text, I hope OpenAI does something distruptively different and releases the weights and trained model. The accidental sentiment neuron is a function of the model, distribution of the input dataset, and the optimizer finding nice saddle points. Insight into these foundational components would make these results…

Post by jcoffland » Thu, Apr 06, 2017, 6:50 PM UTC

aabajian wrote: post #42

I'm trying to understand this statement: "The sentiment neuron within our model can classify reviews as negative or positive, even though the model is trained only to predict the next character in the text." If you look closely at the colorized paragraph in their paper/website, you can see that the major sentiment jumps (e.g. from green to light-green and from light-orangish to red) occur with period characters. Perh…

Post by aabajian » Thu, Apr 06, 2017, 7:01 PM UTC

jcoffland wrote: post #63

aabajian wrote: post #42

I'm trying to understand this statement: "The sentiment neuron within our model can classify reviews as negative or positive, even though the model is trained only to predict the next character in the text." If you look closely at the colorized paragraph in their paper/website, you can see that the major sentiment jumps (e.g. from green to light-green and from light-orangish to red) occur with period characters. Perh…

Why would that matter? Human understanding of sentiment would also go down if you removed vital information such as punctuation.

Post by Houshalter » Thu, Apr 06, 2017, 7:05 PM UTC

d--b wrote: post #66

Can someone explain what is "unsupervised" about this? I'm guessing this is what confuses me most. I think this work is interesting, although when you think about it, it's kind of normal that the model converges to a point where there is a neuron that indicates whether the review is positive or negative. There are probably a lot of other traits that can be found in the "features" layer as well. There are probably neu…

Post by backpropaganda » Thu, Apr 06, 2017, 7:08 PM UTC

emcq wrote: post #22

derefr wrote: post #18

Earlier quoted context omitted.

This actually demonstrates something very interesting, I think: you can take an ML model trained with the "low-level prerequisite knowledge" of a subject, and then very quickly and easily teach it a high-level concept that relies on that knowledge. Which, now that I think about it, makes the human brain and its amazing adaptive general-game-playing abilities a bit less mysterious. Since we humans all have these huge…

What you describe is exactly what practitioners in the field have been doing for years. I think that's why the parent is a bit puzzled at the publication, as it's difficult to understand what's novel.