Principle Component Analysis • Hacker News Board

Re: Principle Component Analysis

#11

Post by hobbyist » Fri, Aug 17, 2012, 9:00 PM UTC

lbarrow wrote: post #5

The author is basically using a linear algebra tool for creating orthogonal basis vectors of a matrix of stock prices. (The PCA is like eigenvector decomposition, but it works on rectangular matrices too. In fact, unlike many operations, it's very fast on unbalanced rectangular matrices!) Since these vectors are, by definition, uncorrelated, they can be very useful in building CAPM-balanced stock portfolios. Using th…

Wow. thts cool really cool. PCA of energy companies being correlated with oil.. intriguing and interesting

Re: Principle Component Analysis

#12

Post by eykanal » Fri, Aug 17, 2012, 9:02 PM UTC

For what it's worth, the best PCA tutorial I've seen online is this blog post, which uses plots to describe the technique:

http://stats.stackexchange.com/a/2700/2019

PCA is nothing more than a "basis shift", or changing where the x and y axes are placed. This image-based tutorial makes understanding very intuitive.

Re: Principle Component Analysis

#13

Post by mturmon » Fri, Aug 17, 2012, 9:02 PM UTC

tel wrote: post #4

PCA goes far deeper than meets the eye. For instance, it's a well-known phenomenon that too much dimensionality can actually drive predictor performance to random, but PCA can mitigate that. It's a basically the bread and butter of practical unsupervised learning.

"bread and butter of practical unsupervised learning" -- true, although I might have said "exploratory data analysis".

If you can make a vector out of it somehow, it can't hurt to try PCA. Because you don't have to figure out some fancy tailored model, or really (cough, cough) understand much about the data at all. (It sounds like I'm being sarcastic, but I'm serious -- sometimes all you want is a quick look.)

Unsupervised clustering is a similar technique.

Re: Principle Component Analysis

#14

Post by apu » Fri, Aug 17, 2012, 9:02 PM UTC

robert00700 wrote: post #3

Nice to see PCA in an HN article, it's a very powerful tool. For those struggling to get the example in this article, I find PCA easier to understand given visual examples, and in less dimensions (try http://en.wikipedia.org/wiki/File:GaussianScatterPCA.png ) Note how this dataset is two dimensional in nature, and PCA yields two vectors. The first gives the direction of the greatest variation, and the next gives the…

FYI Eigenfaces was a ground-breaking theory when introduced...almost 25 years ago. It's no longer used in any serious way for practical face recognition applications.

Re: Principle Component Analysis

#15

Post by j2kun » Fri, Aug 17, 2012, 9:23 PM UTC

robert00700 wrote: post #3

Nice to see PCA in an HN article, it's a very powerful tool. For those struggling to get the example in this article, I find PCA easier to understand given visual examples, and in less dimensions (try http://en.wikipedia.org/wiki/File:GaussianScatterPCA.png ) Note how this dataset is two dimensional in nature, and PCA yields two vectors. The first gives the direction of the greatest variation, and the next gives the…

I wrote a blog post with more detail, and lots of intuitive examples. see http://jeremykun.wordpress.com/2011/07/27/eigenfaces/

Re: Principle Component Analysis

#16

Post by snikolov » Fri, Aug 17, 2012, 9:26 PM UTC

tel wrote: post #4

PCA goes far deeper than meets the eye. For instance, it's a well-known phenomenon that too much dimensionality can actually drive predictor performance to random, but PCA can mitigate that. It's a basically the bread and butter of practical unsupervised learning.

Indeed, projection onto the principal subspace is a kind of regularization, which makes for better generalization.

This kind of makes sense intuitively. These slides go into more detail (the whole course is great), where it says "Projection regularizes!"

http://www.mit.edu/~9.520/spring09/Classes/class07_spectral....

Re: Principle Component Analysis

#17

Post by agentq » Fri, Aug 17, 2012, 10:05 PM UTC

btilly wrote: post #7

PCA is a very useful tool in lots of places. But be warned that when you use it on stocks, you'll find correlations, make your investment, then discover that during a financial crisis all sorts of things that were not previously correlated, now are. Thus your analysis falls apart at exactly the moment you would least want it to do so. Incidentally if you take answers to a wide variety of questions that are meant to t…

In financial practice, asset-level PCA isn't as common, especially in systems where covariance estimation is fraught with misspecification errors. Instead, individual securities first condensed to factors (e.g., for equity some examples are book/price, momentum, large vs. small cap, etc.).

Re: Principle Component Analysis

#18

Post by azmenthe » Fri, Aug 17, 2012, 11:15 PM UTC

agentq wrote: post #17

btilly wrote: post #7

PCA is a very useful tool in lots of places. But be warned that when you use it on stocks, you'll find correlations, make your investment, then discover that during a financial crisis all sorts of things that were not previously correlated, now are. Thus your analysis falls apart at exactly the moment you would least want it to do so. Incidentally if you take answers to a wide variety of questions that are meant to t…

In financial practice, asset-level PCA isn't as common, especially in systems where covariance estimation is fraught with misspecification errors. Instead, individual securities first condensed to factors (e.g., for equity some examples are book/price, momentum, large vs. small cap, etc.).

Yes. The fund I work for has a very successful track record and we take all PCA (on factors) with a huge grain of salt.

Also any strategy that has more than a 20% thesis alignment on PCA (on factors!) is most likely laughed at.

Re: Principle Component Analysis

#19

Post by psb217 » Fri, Aug 17, 2012, 11:25 PM UTC

mturmon wrote: post #8

lbarrow wrote: post #5

The author is basically using a linear algebra tool for creating orthogonal basis vectors of a matrix of stock prices. (The PCA is like eigenvector decomposition, but it works on rectangular matrices too. In fact, unlike many operations, it's very fast on unbalanced rectangular matrices!) Since these vectors are, by definition, uncorrelated, they can be very useful in building CAPM-balanced stock portfolios. Using th…

It's often true that you can figure out what the first handful (say, 3 to 6) PCA components mean, in a large problem. The first is usually the mean of the quantities. It is typical in practice to compute PCA by using the SVD of the data itself; if you subtract the mean first, then of course it will not appear as the first PCA component. In matlab, this is literally a one-liner using the svd of the original data -- no…

FYI, Gabor-like filters pop out from doing ICA (i.e. Independent Components Analysis), not PCA. While PCA looks for orthogonal vectors onto which the data's projection is normally-distributed (among other properties), ICA, roughly speaking, looks for a set of orthogonal vectors onto which the data's projection has maximal kurtosis (among other properties).

It is the kurtosis-maximization of ICA that tends to produce filters mimicking those found in (early layers of) visual cortex. Hence, the production of such filters by techniques like "sparse coding" and "sparse autoencoders", which explicitly pursue highly-kurtotic representations of the training data. PCA, on the other hand, tends to produce checkerboard (i.e. 2d sinusoidal) filters of various frequencies when trained on "natural image patches".

See: "The 'independent components' of natural scenes are edge filters" by Bell and Sejnowski, 1997.

Re: Principle Component Analysis

#20

Post by tel » Fri, Aug 17, 2012, 11:29 PM UTC

mturmon wrote: post #13

tel wrote: post #4

PCA goes far deeper than meets the eye. For instance, it's a well-known phenomenon that too much dimensionality can actually drive predictor performance to random, but PCA can mitigate that. It's a basically the bread and butter of practical unsupervised learning.

"bread and butter of practical unsupervised learning" -- true, although I might have said "exploratory data analysis". If you can make a vector out of it somehow, it can't hurt to try PCA. Because you don't have to figure out some fancy tailored model, or really (cough, cough) understand much about the data at all. (It sounds like I'm being sarcastic, but I'm serious -- sometimes all you want is a quick look.) Unsupe…

I find that more often than expected, PCA (or maybe MDS) gets a majority of the performance of any kind of unsupervised method. If you're really interested in exploring the data and methodologies, then PCA is a poor stopping point... but if you just want it to work, it's surprising how well PCA tradeoffs are good tradeoffs.

All the obvious caveats apply to that whole line of thought, though.