I think one of the most amazing parts of this is how accessible the hardware is right now. You can get world-class AI results with the cost of less than most used cars. In addition, with so many resources freely available through open-source, the ability to get started is very accessible.
Unsupervised sentiment neuron
131–137 of 137 posts
Re: Unsupervised sentiment neuron
#132I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chu…
It's essentially the nlp equivalent transfer learning from the computer vision side.
No this is quite a bit different. In transfer learning in computer vision, people are using a network trained on another, typically large dataset, and using the learned features on a different data set.
This LSTM simply learned to predict the data. It didn't learn some other supervised task.
This would be more similar to autoencoder pretraining, but even that is not quite the same.
Re: Unsupervised sentiment neuron
#133> The model struggles the more the input text diverges from review data
This is where I fear the results will fail to scale. The ability to represent 'sentiment' as one neuron, and its ground truth as uni-dimensional seems most true to corpuses of online reviews where the entire point is to communicate whether you're happy with the thing that came out of the box. Most other forms of writing communicate sentiment in a more multi-dimensional way, and the subject of sentiment is more varied than a single item shipped in a box.
In otherwords, the unreasonable simplicity of modelling a complex feature like sentiment with this method, is something of an artifact of this dataset.
Re: Unsupervised sentiment neuron
#134As far as I understand, it means that there must be a relation between a character's sentiment and what the next character can (/should) be for neural network to use this as a feature, am I right?
Does this mean we have unconsciously developed a language that exposes such relations?
Re: Unsupervised sentiment neuron
#135Earlier quoted context omitted.
Are you in the OMSCS or on-campus program? I'm currently enrolled in CS 6601 (Artificial Intelligence) via OMSCS, and while it's extremely difficult material that requires a lot of personal investment to master, I've found both TAs and other students to be as helpful as one could reasonably expect.
OMSCS. The TAs refuse to answer questions because it could help cheating. They won't review homework because it could help cheating. They've created an environment so full of fear that it's impossible to learn from mistakes. I'm surprised the slack channel is allowed.
I've never had a direct question to a TA go unanswered (and I've asked my fair share of questions). They're not going to just give you the solution to the homework, though. I guess it depends on what kind of questions you're asking?
Re: Unsupervised sentiment neuron
#136As far as I understand, it means that there must be a relation between a character's sentiment and what the next character can (/should) be for neural network to use this as a feature, am I right? Does this mean we have unconsciously developed a language that exposes such relations?
[deleted]
Re: Unsupervised sentiment neuron
#137As far as I understand, it means that there must be a relation between a character's sentiment and what the next character can (/should) be for neural network to use this as a feature, am I right? Does this mean we have unconsciously developed a language that exposes such relations?
They muse about the reason behind the sentiment neuron in the paper.
"It is an open question why our model recovers the concept of sentiment in such a precise, disentangled, interpretable, and manipulable way. It is possible that sentiment as a conditioning feature has strong predictive capability for language modelling. This is likely since sentiment is such an important component of a review."
They go on to frame that as an important consideration for further work like this:
"Our work highlights the sensitivity of learned representations to the data distribution they are trained on. The results make clear that it is unrealistic to expect a model trained on a corpus of books, where the two most common genres are Romance and Fantasy, to learn an encoding which preserves the exact sentiment of a review."
I'm wondering if a "funniness" neuron could be discovered in a model trained on millions of jokes of various funniness, or what sorts of undiscovered meaning there is in other neurons in this model.