Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural Networks, Manifolds, and Topology

colah.github.io

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Neural Networks, Manifolds, and Topology

#21

Post by GrantS »

ArbitraryLimits wrote: post #18

Wouldn't this only apply to neural networks as used for classification? I mean the general paradigm of deforming curves until they're separated by a hyperplane seems pretty obvious now that I see it in front of me, but what about neural networks used to approximate continuous functions?

I'll take a stab at this (I'm a decade out from my last machine learning class, so no guarantees on correctness): The only reason it's fitting a hyperplane is because one class is being mapped to the continuous value -1.0 and the other class is being mapped to the continuous value 1.0 and there's a thresholding step (the hyperplane perpendicular to the line onto which the continuous values are being projected) at the end to determine the class. If you're doing regression instead of classification, your training data will be fed in with more output values than just 1.0 and -1.0 and you'll omit the thresholding at the end, but otherwise the behavior and intuition should be the same as in the article.

Re: Neural Networks, Manifolds, and Topology

#22

Post by ot »

ejenk wrote: post #17
ot wrote: post #5

This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs. In particular this: > (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.) Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but ever…

I think the author has confused NP with NP-hard (or NP-complete). Indeed, it is extremely unlikely that the unknottedness problem is NP-complete, as it is already in NP ∩ co-NP (assuming the generalized Riemann hypothesis). Furthermore, we know that the problem of computing many knot invariants is "only" BQP-complete, so it's possible that unknottedness is in BQP (i.e., efficiently solvable by quantum computers), tho…

Indeed I read that as NP-complete. That's very interesting, do you have any references on it? Even if knots are ruled out, there might be hard topological decision problems that can be reduced to linear separability under homeomorphism, maybe in higher dimension.

Re: Neural Networks, Manifolds, and Topology

#23

Post by tgflynn »

ot wrote: post #5

This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs. In particular this: > (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.) Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but ever…

I think machine learning boils down to 2 fundamental issues:

1) Having a model class that is powerful enough to represent the data.

and

2) Being able to find a "good enough" optimal solution over that model class.

In principle over-fitting can be addressed through the choice of the particular optimization criteria in (2).

Neural networks are used because they are powerful enough to represent many data sets of interest and because there exist good algorithms for finding LOCAL optima.

Unfortunately they are complex non-linear, non-convex functions and hence finding a global optimum is most likely NP-hard.

We are left then with the heuristic of hoping that the local optima we are able to find are "close enough" to the global optimum to meet our needs.

This seems to work reasonably well for certain types of problems, like image classification, where the underlying data possess a certain simplicity or "smoothness", but less well for other problems, like logic problems where good and bad solutions may not be "near by" in any sense that the local optimization algorithms are able to discover.

Re: Neural Networks, Manifolds, and Topology

#25

Post by yummyfajitas »

I don't like his proof that a 1-2 hidden unit system can't handle the red circle in the blue torus. Feels too handwavy.

My proof: Suppose the matrix W is singular. Then there exists a nonzero vector v for which W v = 0. Find a point x in the red circle, then find a different point y=x+av (for some scalar a) in the blue torus. Wy+b=W(x+av)+b=Wx+aWv+b=Wx+a0+b=wX+b. Fin.

Re: Neural Networks, Manifolds, and Topology

#27

Post by rdtsc »

reader5000 wrote: post #14
rdtsc wrote: post #13

It is always good to reference SVMs ( http://en.wikipedia.org/wiki/Support_vector_machine ) when talking about neural networks. The advantage of SVMs is that after re-mapping the data so it becomes separable, they also guarantee to separate it as clear as possible. Not sure if they can solve all the type of problems as well but in some cases it is considered a better, more analytic approach.

"after re-mapping the data so it becomes separable" This is sort of the hard part.

Not saying it isn't that is the main bulk of the algorithm, just saying it there is an additional benefit of SVMs in the end.

Re: Neural Networks, Manifolds, and Topology

#28

Post by fear91 »

rdtsc wrote: post #27
reader5000 wrote: post #14

Earlier quoted context omitted.

"after re-mapping the data so it becomes separable" This is sort of the hard part.

Not saying it isn't that is the main bulk of the algorithm, just saying it there is an additional benefit of SVMs in the end.

SVM is great, it's often stacked on top of neural networks to do classification.

But it can't really deal with extracting features, especially from things like images and sounds. If you fed images ( pixel by pixel ) straight to the SVM and tell it to classify them, it would do extremely poorly - especially with big images.

A better thing is to have a deep ( layered ) network that will get pixel data from the input and pass it through layers that will then extract some useful features from it - for example, learn to recognize lines.

Then, after this information passes through the whole network, you are finished with smaller amount of features that are more useful for classification. Then you an send this information to SVM and it will deal with it a lot easier than if you tried to feed it straight pixels.

The thing is, we don't really have problems with classification algorithms - SVM does extremely well. What is the biggest hurdle is simply extracting features that then can be fed to well-working classification algorithms.

Re: Neural Networks, Manifolds, and Topology

#29

Post by derf_ »

> One could learn the vector field at fixed points (just take some fixed points from the training set to use as anchors) and interpolate in some manner.

One such interpolation that has nice properties (smooth everywhere, invertible, can be constructed for any set of pointwise correspondences) are diffeomorphisms.

Unfortunately, the mathematical objects are infinite-dimensional, which makes computation with them somewhat involved, even if they are derived from a small-dimensional set of point correspondences. See, e.g., " rel="nofollow">http://www.cs.unc.edu/Research/MIDAG/pubs/papers/Joshi_TIP_2... for one method of constructing them.

Re: Neural Networks, Manifolds, and Topology

#30

Post by sushirain »

The article is exceptionally clear and well presented. We need more of this level of writing about machine learning.

For me, it is most important to note what the article touches only at the end. If you have good features and representation, a linear model may be enough. Real world data and tasks usually don't require knot untying. In my opinion, the model (e.g., neural net) should be kept simple, and the challenge is in finding the right features and representation.

Coates & Ng (2012) wrote:

Many algorithms are available to learn deep hierarchies of features from unlabeled data, especially images. In many cases, these algorithms involve multi-layered networks of features (e.g., neural net- works) that are sometimes tricky to train and tune and are difficult to scale up to many machines effectively. Recently, it has been found that K-means clustering can be used as a fast alternative training method. The main advantage of this approach is that it is very fast and easily implemented at large scale. On the other hand, employing this method in practice is not completely trivial: K-means has several limitations, and care must be taken to combine the right ingredients to get the system to work well.

http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.p...

Post reply on HN