Decoding the Thought Vector • Hacker News Board

Re: Decoding the Thought Vector

#21

Post by antognini » Wed, Nov 30, 2016, 11:01 PM UTC

sweetdreamerit wrote: post #15

numinary1 wrote: post #6

When I was working on a recommender for television shows, I ran SVD on a large User/Item matrix to create a low rank approximation, essentially reducing thousands of user features (TV show preferences) to user vectors representing twenty or thirty abstract "features". Then I looked at the actual item preferences of users who expressed each feature at the greatest and least magnitude. The features, in some cases, mapp…

A question: is this much better / different than a principal component analysis (or a factor analysis)?

It's a bit of an apples/oranges comparison to compare SVD to PCA. SVD is a numerical technique, whereas PCA is a method to analyze a dataset. You can use SVD to perform PCA (although there are other ways to perform PCA without explicitly doing a SVD). I'm guessing that the GP performed PCA using SVD. There's a good Stack Exchange answer to exactly this question here:

http://stats.stackexchange.com/questions/121162/is-there-any...

Re: Decoding the Thought Vector

#22

Post by jmde » Wed, Nov 30, 2016, 11:23 PM UTC

numinary1 wrote: post #6

When I was working on a recommender for television shows, I ran SVD on a large User/Item matrix to create a low rank approximation, essentially reducing thousands of user features (TV show preferences) to user vectors representing twenty or thirty abstract "features". Then I looked at the actual item preferences of users who expressed each feature at the greatest and least magnitude. The features, in some cases, mapp…

I started reading the essay not knowing what to think, and it turned out to be more relevant to my work than I thought.

The issues being discussed in the essay have been a central issue in some area of psychology and behavioral sciences for some time--how to interpret components such as these.

One thought about your "coming into focus at a certain level of compression" comment: I've done some analyses of these vectors as applied to text samples, and one thing that struck me was how unreplicable some of them were across datasets that should be ostensibly similar (but are not the same). Others, in contrast, reappeared across multiple corpora. To the extent some of these components represent "real" features, they should reappear consistently across different datasets where you'd expect them to. That is, they should be robust to changes in idiosyncratic features of the database.

Re: Decoding the Thought Vector

#23

Post by antognini » Wed, Nov 30, 2016, 11:27 PM UTC

mooneater wrote: post #19

I find it counter-intuitive that thought vectors in should have "Linear Structure" in multilevel autoencoders. Since the whole appeal of neural networks is that they can model non-linear functions. Why would the autoencoder end up with an encoding that is essentially linear?

The goal of a neural network is to take a complicated manifold and, through each of its layers, flatten it out into progressively more and more linear manifolds. If you are building a classifier, then the inputs to the last layer will necessarily have to be linearly separable because the final layer is essentially linear --- the softmax operation just transforms the pre-activation values from logits to probabilities.

So if the NN is well trained, the second to last layer will have linearly separated the different classes (as much as possible, anyway). Earlier layers may not have completely linearly separated their inputs, but they are probably going to lie along simpler manifolds than even earlier inputs.

There's a good blog post on this here: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

Re: Decoding the Thought Vector

#24

Post by mooneater » Fri, Dec 02, 2016, 1:17 AM UTC

antognini wrote: post #23

mooneater wrote: post #19

I find it counter-intuitive that thought vectors in should have "Linear Structure" in multilevel autoencoders. Since the whole appeal of neural networks is that they can model non-linear functions. Why would the autoencoder end up with an encoding that is essentially linear?

The goal of a neural network is to take a complicated manifold and, through each of its layers, flatten it out into progressively more and more linear manifolds. If you are building a classifier, then the inputs to the last layer will necessarily have to be linearly separable because the final layer is essentially linear --- the softmax operation just transforms the pre-activation values from logits to probabilities.…

Yes I understand that conceptually. But the middle autoencoder layer is not really an output layer, what would constrain it to linear representations? I assume normal output layers are constrained to linear representations by the supervised training process.