Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unsupervised sentiment neuron

blog.openai.com

Post reply on HN
Print view View original

91–100 of 137 posts

Re: Unsupervised sentiment neuron

#91

Post by jorgemf »

1024core wrote: post #4

I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…

isn't it impressive? They trained a network to predict the next character and when the finished they trained the last layer to predict sentiment with only few examples. The network was able to learn sentiment somehow in the first training phase without telling it!

For me this open my mind to new opportunities when training deep learning. For example I can do the same for images: train a network to recognize objects and later use the same network to predict sentiment or prettiness for example. And the best thing is that I don't need a lot of labeled examples for the second phase of training!

Re: Unsupervised sentiment neuron

#92

Post by technotony »

derefr wrote: post #18
1024core wrote: post #4

I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…

This actually demonstrates something very interesting, I think: you can take an ML model trained with the "low-level prerequisite knowledge" of a subject, and then very quickly and easily teach it a high-level concept that relies on that knowledge. Which, now that I think about it, makes the human brain and its amazing adaptive general-game-playing abilities a bit less mysterious. Since we humans all have these huge…

What you are proposing with the rat brain simulation/scan seems pretty close to what the Blue Brain Project folks are doing: http://bluebrain.epfl.ch/page-56882-en.html

Re: Unsupervised sentiment neuron

#93

Post by technotony »

gdb wrote: post #11
joshuamorton wrote: post #10

Earlier quoted context omitted.

To further clarify: does unlabeled mean "we didn't use sentiment data" or "we were only trying to predict the next character given the prior characters", since the amazon data does come with associated 1-5 star ratings, were those used or not?

We did not use the star ratings.

Have you looked at cross referencing against the star ratings? Would be interesting to see how predictive this was of those ratings... or even if another variable in the system could predict that?

Re: Unsupervised sentiment neuron

#94

Post by 013a »

virmundi wrote: post #71
ericjang wrote: post #38

Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate…

From my little experience with the AI community, I think people in it love to obfuscate things. Any attempt to make a topic approachable, even if some of the details are lost, get smacked around. I face this every day in my Masters. If you don't already come with a knowledge of AI + Stats, you're on your own. The community, including the teachers, don't want to teach the mundane.

Techy people, by and large, don't understand marketing. We think that technology should sell itself, and anyone who needs convincing of the superiority of some solution is just dumber than they are. Combined with the elitism complex we see all over academia and the genuine complexity of AI research as it stands today... recipe for sociopathic disaster.

Re: Unsupervised sentiment neuron

#95

Post by fiter »

wackspurt wrote: post #87
fiter wrote: post #74

Earlier quoted context omitted.

I wouldn't expect that the neurons are orthogonal on a set of features which we find interesting (sentiment, geographical location). They could be bound up in some other basis of features that we do not find interesting. Other people do not expect this because there are papers about how to incentivize neurons to correspond to interesting features.

>> Other people do not expect this because there are papers about how to incentivize neurons to correspond to interesting features. Could you clarify that statement? Are you saying that it was unusual for this group to find such a neuron? Also, I did not know that there are papers on how to incentivize neurons to correspond to interesting features. Could you please give me some references on those?

The paper I was thinking of is called: "InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets"[0]. I do not have experience training and investigating neural nets, but from what I read in that paper, there's no reason to presume you'll find neurons that represent a feature you're interested in. In the paper they alter the reward function to get neurons that correspond to the features they are interested in.

[0] https://arxiv.org/pdf/1606.03657v1.pdf

Re: Unsupervised sentiment neuron

#97

Post by fiter »

eanzenberg wrote: post #90
ericjang wrote: post #38

Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate…

Is it wrong to be critical of research? Back in my previous life of doing basic research I scrutinized papers left and right. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ towards the end has similar methodology and is 1.5 years old. Hype is an interesting thing especially when it comes from laymen.

If they're getting better results than the previous state of the art, I think the most important point of this research is its critique of previous papers! The previous state of the art research needs to consider such old, basic techniques to improve their own results.

Re: Unsupervised sentiment neuron

#98

Post by laingc »

eanzenberg wrote: post #90
ericjang wrote: post #38

Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate…

Is it wrong to be critical of research? Back in my previous life of doing basic research I scrutinized papers left and right. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ towards the end has similar methodology and is 1.5 years old. Hype is an interesting thing especially when it comes from laymen.

As someone familiar with the field, you likely know this already, but the similarities between the Karpathy post from 2015 and this work from OpenAI is likely because Karpathy is a founder and lead researcher at OpenAI.

Re: Unsupervised sentiment neuron

#99

Post by nl »

rspeer wrote: post #35
nl wrote: post #3

So char-by-char models is the next Word2Vec then. Pretty impressive results. It would be interesting to see how it performed for other NLP tasks. I'd be pretty interested to see how many neurons it uses to attempt something like stance detection. Data-parallelism was used across 4 Pascal Titan X gpus to speed up training and increase effective memory size. Training took approximately one month. Everytime I look at so…

Yeah, part of what let word2vec make such a splash that it became the one word embedding model everyone has heard of, is that the word2vec team released their model. This is a really cool example OpenAI has, but I don't know why I should ultimately care about their character model more than anyone else's if all we've got is their description of how cool it is. I hope OpenAI defies their reputation for closedness and…

I don't know why I should ultimately care about their character model more than anyone else's if all we've got is their description of how cool it is.

Well an unsupervised technique that learns this much meaning from text is amazing! I meant it when I said this might supplement word2vec, and that would make it one of the most important breakthroughs in years.

The comments critical of OpenAI don't make a lot of sense. They have always been very good at releasing stuff, and my comment about waiting for a trained model should be read as jealousy over not being able to train it myself..

Re: Unsupervised sentiment neuron

#100

Post by srush »

If you are interested in looking at the model in more detail, we (@harvardnlp) have uploaded the model features to LSTMVis [1]. We ran their code on amazon reviews and are showing a subset of the learned features. Haven't had a chance to look further yet, but it is interesting to play with.

[1] http://lstm.seas.harvard.edu/client/pattern_finder.html?data...

Post reply on HN