I'm familiar with this idea - but it's nice to see explained with cute little puppies and kittens.
The Curse of Dimensionality in Classification
21–27 of 27 posts
Re: The Curse of Dimensionality in Classification
#22Earlier quoted context omitted.
Cross-validation (actually, this is mentioned toward the end of the article). Basically, fit the the classifier with a subset of the data and test the predictions on the remainder. Predictions for out-of-sample data will be poor if you have overfitting. http://en.wikipedia.org/wiki/Cross-validation_%28statistics%...
yeah, this one is quite intuitive, but it reduces the training sample size.
Once you find the optimal parameters, you can then train the model again on the entire dataset.
Re: The Curse of Dimensionality in Classification
#23is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.
Cross-validation, separate training and test data sets, or AIC/BIC (AIC is more forgiving than BIC) if you can get a reasonable estimate of your "degrees of freedom". (For many models, however, d.f. is either not defined or intractable. For bagged or boosted trees, for example, you need CV or a test set.) If you're data rich, you tend not to use CV but to have two or three sets. The reason 3 is better is because you…
What do you do for a living?
Re: The Curse of Dimensionality in Classification
#24Can someone explain to me Figure 6 please? How does projecting the 3d space (and 2d green plane) lead to the regions around the cat's heads?
Re: The Curse of Dimensionality in Classification
#25Can someone explain to me Figure 6 please? How does projecting the 3d space (and 2d green plane) lead to the regions around the cat's heads?
I will try:
Imagine flattening the 3D space (Figure 5) down onto the 2D space of the floor.
Then, the image is trying to illustrate which sections of the 2D space have been selected as 'cat' by the classifier (green plane).
Visually, you can imagine that the cat heads have each pushed down a chunk of the green plane (the dog heads have held the green plane up off the ground).
This illustrates that the third dimension was important for separating out the cats: they certainly aren't separated by a clean line in those two dimensions.
However, it also illustrates that the separation might be slightly contrived: looked at in 2D, the green plane seems to have plucked out the odd cats correctly but without logic.
One goal is to show that using a high number of dimensions will guarantee that you can separate dogs and cats, but that this is just an over-fitted solution to the data set that you have: it will not continue to work when you apply it to further data.
Re: The Curse of Dimensionality in Classification
#26Can someone explain to me Figure 6 please? How does projecting the 3d space (and 2d green plane) lead to the regions around the cat's heads?
I will try: Imagine flattening the 3D space (Figure 5) down onto the 2D space of the floor. Then, the image is trying to illustrate which sections of the 2D space have been selected as 'cat' by the classifier (green plane). Visually, you can imagine that the cat heads have each pushed down a chunk of the green plane (the dog heads have held the green plane up off the ground). This illustrates that the third dimension…
> Visually, you can imagine that the cat heads have each pushed down a chunk of the green plane (the dog heads have held the green plane up off the ground).
Thank you, that helps. I was mentally projecting the whole plane down and could see why it wasn't all-green (or all-not).
Re: The Curse of Dimensionality in Classification
#27Another good dimensionality reduction technique to consider is Latent Dirichlet Allocation. I use this approach for natural language or other "bursty" data sets. "Bursty" data sets are characterized by having Zipfian distribution over features, but certain long-tail features achieving a higher probability of multiple observations given initial observation in an instance. For example, "armadillo" is relatively rare, b…
Can you provide a link to an article discussing how you can treat the latent characteristics as a point in Euclidean space?
Here's some basic background: http://faculty.washington.edu/jwilker/559/SteyversGriffiths....