Unsupervised sentiment neuron
blog.openai.com
Unsupervised sentiment neuron
1–10 of 137 posts
Re: Unsupervised sentiment neuron
#2The synthetic text they generated was surprisingly realistic, despite being generic.
If I were perusing a dozen reviews I probably wouldn't have spotted the AI-generated ones in the crowd.
Re: Unsupervised sentiment neuron
#3So char-by-char models is the next Word2Vec then. Pretty impressive results.
It would be interesting to see how it performed for other NLP tasks. I'd be pretty interested to see how many neurons it uses to attempt something like stance detection.
Data-parallelism was used across 4 Pascal Titan X gpus to speed up training and increase effective memory size. Training took approximately one month.
Everytime I look at something like this I find a line like that and go: "ok that's ncie.. I'll wait for the trained model".
Re: Unsupervised sentiment neuron
#4I don't know, but this seems a bit hyped in places.
They start with:
> Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples.
Hmm, that sounds pretty impressive. But then later you read:
> We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chunk of text. Training took one month across four NVIDIA Pascal GPUs
Wait, what? How did "232 examples" transform into "82 million"??
OK, I get it: they pretrained the network on the 82M reviews, and then trained the last layer to do the sentiment analysis. But you can't honestly claim that you did great with just 232 examples!
Re: Unsupervised sentiment neuron
#5moved that needle I guess
Re: Unsupervised sentiment neuron
#6I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
Thanks for the feedback — added context to that sentence to make it more clear!
The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.
Re: Unsupervised sentiment neuron
#7Impressive the abstraction NNs can achieve from just character prediction. Do the other systems they compare to also use 81M Amazon reviews for training? Seems disingenuous to claim "state-of-the-art" and "less data" if they haven't.
Re: Unsupervised sentiment neuron
#8[deleted]
Re: Unsupervised sentiment neuron
#9I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
Thanks for the feedback — added context to that sentence to make it more clear! The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.
Very interesting, this reminds me of the 2012 paper by Andrew Ng:
Building High-level Features Using Large Scale Unsupervised Learning
Re: Unsupervised sentiment neuron
#10I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
Thanks for the feedback — added context to that sentence to make it more clear! The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.
To further clarify: does unlabeled mean "we didn't use sentiment data" or "we were only trying to predict the next character given the prior characters", since the amazon data does come with associated 1-5 star ratings, were those used or not?