Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Curse of Dimensionality in Classification

visiondummy.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: The Curse of Dimensionality in Classification

#11

Post by Bill_Dimm »

nraynaud wrote: post #7

is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.

Cross-validation (actually, this is mentioned toward the end of the article). Basically, fit the the classifier with a subset of the data and test the predictions on the remainder. Predictions for out-of-sample data will be poor if you have overfitting.

http://en.wikipedia.org/wiki/Cross-validation_%28statistics%...

Re: The Curse of Dimensionality in Classification

#12

Post by amitdeshwar »

languagehacker wrote: post #8

Another good dimensionality reduction technique to consider is Latent Dirichlet Allocation. I use this approach for natural language or other "bursty" data sets. "Bursty" data sets are characterized by having Zipfian distribution over features, but certain long-tail features achieving a higher probability of multiple observations given initial observation in an instance. For example, "armadillo" is relatively rare, b…

Can you provide a link to an article discussing how you can treat the latent characteristics as a point in Euclidean space?

Re: The Curse of Dimensionality in Classification

#13

Post by fiatmoney »

languagehacker wrote: post #8

Another good dimensionality reduction technique to consider is Latent Dirichlet Allocation. I use this approach for natural language or other "bursty" data sets. "Bursty" data sets are characterized by having Zipfian distribution over features, but certain long-tail features achieving a higher probability of multiple observations given initial observation in an instance. For example, "armadillo" is relatively rare, b…

If you're explicitly looking to calculate document similarity via cosine distance, you may also want to try a technique that explicitly tries to map into an orthogonal space, like any of the principal components analysis variants.

Re: The Curse of Dimensionality in Classification

#14

Post by ced »

kmike84 wrote: post #9
cortexman wrote: post #3

Earlier quoted context omitted.

Not really. The article mentions that using linear methods (i.e., LIBLINEAR) is one way to avoid the curse. LIBLINEAR is specifically designed for situations in which you have many features and relatively few training instances. When using a linear classifier it may make sense to simply generate as many features as you can, and then use, i.e., lasso regression in order to do feature selection. http://www.csie.ntu.edu…

But the article used a linear model to demonstrate the curse, and the model was overfit just with 3 dimensions. There is clearly something missing: for example, for text data it is not uncommon to have thousands or hundred thousands of dimensions, and algorithms work fine. I think the missing piece is regularisation. It doesn't have to do feature selection and actually reduce the number of dimensions, but you're righ…

The article had very few data points, that's why it worked with 3 dimensions. The deciding factor is how N (effective number of data points) compares with p (effective number of features).

Re: The Curse of Dimensionality in Classification

#15

Post by christopheraden »

RK wrote: post #10
nraynaud wrote: post #7

is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.

You could make a plot like Figure 1. Look for the turning point (do some calculus if you can, i.e. d(perf)/d(dim) = 0).

Derivatives require continuity. It would be sufficient to simply look at which number of dimensions gave you the best cross-validated classification rate.

Re: The Curse of Dimensionality in Classification

#16

Post by christopheraden »

nraynaud wrote: post #7

is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.

There's some additional information you'd need to determine this. Suppose, for the sake of argument, I only have one feature, X. Pretend I extend it into two dimensions by simply replicating the feature. In two dimensions, (X,X) forms a perfect straight line, which should make it clear that using an additional dimension didn't gain you anything.

Testing whether or not to include additional terms requires an understanding of the distribution of the response, as well as the amount of collinearity with the features (how similar the features are). There are some ways to do this in statistics, but this is more of something they do in inference as opposed to prediction.

Heuristically, the most common way is just to look at the cross-validated classification error and compare it with and without a feature (or set of features) in question. Asking about the distribution of the cross-validated classification rate is an interesting statistical question, though!

Re: The Curse of Dimensionality in Classification

#17

Post by nraynaud »

Bill_Dimm wrote: post #11
nraynaud wrote: post #7

is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.

Cross-validation (actually, this is mentioned toward the end of the article). Basically, fit the the classifier with a subset of the data and test the predictions on the remainder. Predictions for out-of-sample data will be poor if you have overfitting. http://en.wikipedia.org/wiki/Cross-validation_%28statistics%...

yeah, this one is quite intuitive, but it reduces the training sample size.

Re: The Curse of Dimensionality in Classification

#18

Post by michaelochurch »

nraynaud wrote: post #7

is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.

Cross-validation, separate training and test data sets, or AIC/BIC (AIC is more forgiving than BIC) if you can get a reasonable estimate of your "degrees of freedom". (For many models, however, d.f. is either not defined or intractable. For bagged or boosted trees, for example, you need CV or a test set.)

If you're data rich, you tend not to use CV but to have two or three sets. The reason 3 is better is because you ideally have (a) a training set for building models with known, fixed "hyperparameters" (e.g. regularization coefficients, tree sizes, neural net topologies), (b) a validation set for evaluating models with varying hyperparameters, in order to optimally select them, and (c) a test set on which you can evaluate the model for accuracy after your hyperparameters are chosen from b. Cross-validation is typically what you need to do when you have a small number of observations (say, 1000).

Re: The Curse of Dimensionality in Classification

#19

Post by apu »

alceufc wrote: post #6

I really like the idea of the Web site as a whole: explaining concepts from computer vision in a simple way. When I was starting my masters course I was interested in learning what the concept of bag of words in computer vision was all about. Although it is straightforward technique, there are few examples on the Web explaining how to implement it (clustering the feature vectors and etc.)

Although not widely known, the best way to learn about specific computer vision topics in detail is usually through the websites for tutorials held at the premiere computer vision conferences. These are CVPR, ICCV, and ECCV. For example, here are the tutorials held at ICCV 2013: http://www.iccv2013.org/tutorials.php

If you click through, you'll see that most of them have links to slides, and some even have video coverage and/or links to software as well. They're also usually presented by experts in that area.

Finally, note that even if the most recent conferences don't have relevant tutorials for what you're looking for, you can still often find good material by going back further in time (e.g., tutorials presented 2 or more years ago). Although the state of the art may have advanced since then, the fundamentals usually don't change very frequently.

Re: The Curse of Dimensionality in Classification

#20

Post by mturmon »

RK wrote: post #10
nraynaud wrote: post #7

is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.

You could make a plot like Figure 1. Look for the turning point (do some calculus if you can, i.e. d(perf)/d(dim) = 0).

Alas, it rarely looks clean like that. Actually never, in my experience, for real problems.
Post reply on HN