Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural Networks, Manifolds, and Topology

colah.github.io

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Neural Networks, Manifolds, and Topology

#11

Post by gajomi »

ot wrote: post #5

This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs. In particular this: > (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.) Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but ever…

There is large body of work from the mid 1980s and early 1990s that addresses the question of hardness, sensitivity and robustness from various statistical physics/computer science collaborations. Of course, it is still ongoing, but that was a big period. The basic thrust of this program is to take methods from statistical mechanics and use them calculate not just worst case complexity, but also average case, best case, and everything in between. It is hard to think of a reference to end all references, but as far as books go you might want to check out: http://www.amazon.com/The-Nature-Computation-Cristopher-Moor.... For an article that briefly addresses some results related to kSAT there is: http://www.cs.cornell.edu/selman/papers/pdf/99.nature.phase.....

Re: Neural Networks, Manifolds, and Topology

#12

Post by tlarkworthy »

ot wrote: post #5

This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs. In particular this: > (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.) Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but ever…

I have seen Riemann geometry applied to neural networks. Basically the state space is curved, so you can speed up learning if take the curvature into account when sampling in the space of possible neural nets. (you want to explore the hypothesis space evenly)

Information geometry for Neural networks, 1998, Daniel Wagenaar:-

http://www.danielwagenaar.net/res/papers/98-Wage2.pdf

Re: Neural Networks, Manifolds, and Topology

#13

Post by rdtsc »

It is always good to reference SVMs (http://en.wikipedia.org/wiki/Support_vector_machine) when talking about neural networks.

The advantage of SVMs is that after re-mapping the data so it becomes separable, they also guarantee to separate it as clear as possible.

Not sure if they can solve all the type of problems as well but in some cases it is considered a better, more analytic approach.

Re: Neural Networks, Manifolds, and Topology

#14

Post by reader5000 »

rdtsc wrote: post #13

It is always good to reference SVMs ( http://en.wikipedia.org/wiki/Support_vector_machine ) when talking about neural networks. The advantage of SVMs is that after re-mapping the data so it becomes separable, they also guarantee to separate it as clear as possible. Not sure if they can solve all the type of problems as well but in some cases it is considered a better, more analytic approach.

"after re-mapping the data so it becomes separable" This is sort of the hard part.

Re: Neural Networks, Manifolds, and Topology

#15

Post by xcthulhu »

TomAnthony wrote: post #6

A bit of an aside: Are there any methods for compressing/collapsing/simplifying a neural network. What I mean is imagine you've built and trained a neural network, as per the article, it is hard to ascertain exactly what it is doing. I was wondering whether there is work in this area, and it occurred to me a possible first step would be to collapse the neural network to a simpler but functionally equivalent structure…

The Universal Approximation Theorem[1] asserts that you only ever need one hidden layer, which at least asserts that "an (approximate) simplification exists".

But I can't say off the top of my head how you'd collapse an ANN just two hidden layers into 1. It's not obvious how sigmoid functions compose, but I suppose I should give it more thought...

---------- 1: http://en.wikipedia.org/wiki/Universal_approximation_theorem

Re: Neural Networks, Manifolds, and Topology

#16

Post by michaelochurch »

xcthulhu wrote: post #15
TomAnthony wrote: post #6

A bit of an aside: Are there any methods for compressing/collapsing/simplifying a neural network. What I mean is imagine you've built and trained a neural network, as per the article, it is hard to ascertain exactly what it is doing. I was wondering whether there is work in this area, and it occurred to me a possible first step would be to collapse the neural network to a simpler but functionally equivalent structure…

The Universal Approximation Theorem[1] asserts that you only ever need one hidden layer, which at least asserts that "an (approximate) simplification exists". But I can't say off the top of my head how you'd collapse an ANN just two hidden layers into 1. It's not obvious how sigmoid functions compose, but I suppose I should give it more thought... ---------- 1: http://en.wikipedia.org/wiki/Universal_approximation_the…

Although one hidden layer suffices theoretically, most of the cutting-edge work involves multilayer networks. Perhaps paradoxically, those become more "auditable" from what I've heard. (I've spoken to people with real-world experience of applying ANNs to unsolved problems, but I am not one.) What I've heard is that, with one layer, you get too much overloading (i.e. different stuff ending up at the same hidden node) to understand what it is doing, but that deeper networks are more legible (with appropriate visualization tools).

UAT says that a solution exists, but it doesn't put a limit on the number of nodes required, so it would have you doing an optimization in a space that is not just large, but of arbitrary finite dimension. It can be pretty nonconstructive (in the sense of proving "there exists" without showing how to find something) insofar as it's often non-trivial to get convergence to a working solution in reasonable time.

As for how sigmoids compose, imagine how bell-shaped curves would compose, just as you can make a painting out of bell-shaped "points" if allowed arbitrary precision/steepness. Now, the difference of two sigmoids can be bell-shaped, e.g. http://www.wolframalpha.com/input/?i=plot+y+%3D+1%2F%281%2Be... . I don't know how much this means in practice, but it establishes the possibility.

Re: Neural Networks, Manifolds, and Topology

#17

Post by ejenk »

ot wrote: post #5

This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs. In particular this: > (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.) Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but ever…

I think the author has confused NP with NP-hard (or NP-complete). Indeed, it is extremely unlikely that the unknottedness problem is NP-complete, as it is already in NP ∩ co-NP (assuming the generalized Riemann hypothesis). Furthermore, we know that the problem of computing many knot invariants is "only" BQP-complete, so it's possible that unknottedness is in BQP (i.e., efficiently solvable by quantum computers), though I don't know the expert opinion on this.

Re: Neural Networks, Manifolds, and Topology

#18

Post by ArbitraryLimits »

Wouldn't this only apply to neural networks as used for classification? I mean the general paradigm of deforming curves until they're separated by a hyperplane seems pretty obvious now that I see it in front of me, but what about neural networks used to approximate continuous functions?

Re: Neural Networks, Manifolds, and Topology

#19

Post by agibsonccc »

michaelochurch wrote: post #16
xcthulhu wrote: post #15

Earlier quoted context omitted.

The Universal Approximation Theorem[1] asserts that you only ever need one hidden layer, which at least asserts that "an (approximate) simplification exists". But I can't say off the top of my head how you'd collapse an ANN just two hidden layers into 1. It's not obvious how sigmoid functions compose, but I suppose I should give it more thought... ---------- 1: http://en.wikipedia.org/wiki/Universal_approximation_the…

Although one hidden layer suffices theoretically, most of the cutting-edge work involves multilayer networks. Perhaps paradoxically, those become more "auditable" from what I've heard. (I've spoken to people with real-world experience of applying ANNs to unsolved problems, but I am not one.) What I've heard is that, with one layer, you get too much overloading (i.e. different stuff ending up at the same hidden node)…

With respect to the deep learning networks ( as well as more traditional with just a weight matrix and bias), we can look to matrices for this. I will offer a compressed representation, not so much a way of pruning, but I explain why below.

Each single layer neural network is made up of 3 matrices, a weight matrix (connections), visible bias, and a hidden bias.

In theory, this can be represented as a flattened array.

This is what I do in deeplearning4j[1] for optimization (note: I'm the author)

The problem with pruning neural nets so to speak is this isn't really a search problem we're solving like A*. Both are graphs in a sense, but each neuron in a neural net when we have backpropagation gets updated to correct for error that it caused, rather than pruning like in Alpha Beta Pruning for Game playing AI.

I will offer one last thing and say that the way neural nets learn (especially if you stream data in to it for training rather than training all at once via online/mini batch learning) each neuron also tends to learn different components of an overall solution and some will activate more on certain feature vectors when you train them on an overall data set.

A solution to this is to set the neurons relative to the input size[2].

[1] https://deeplearning4j.org/

[2] http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf

Re: Neural Networks, Manifolds, and Topology

#20

Post by [deleted] »

michaelochurch wrote: post #16
xcthulhu wrote: post #15

Earlier quoted context omitted.

The Universal Approximation Theorem[1] asserts that you only ever need one hidden layer, which at least asserts that "an (approximate) simplification exists". But I can't say off the top of my head how you'd collapse an ANN just two hidden layers into 1. It's not obvious how sigmoid functions compose, but I suppose I should give it more thought... ---------- 1: http://en.wikipedia.org/wiki/Universal_approximation_the…

Although one hidden layer suffices theoretically, most of the cutting-edge work involves multilayer networks. Perhaps paradoxically, those become more "auditable" from what I've heard. (I've spoken to people with real-world experience of applying ANNs to unsolved problems, but I am not one.) What I've heard is that, with one layer, you get too much overloading (i.e. different stuff ending up at the same hidden node)…

[deleted]
Post reply on HN