Neural Networks, Manifolds, and Topology
colah.github.io
Neural Networks, Manifolds, and Topology
1–10 of 31 posts
Re: Neural Networks, Manifolds, and Topology
#2This is very interesting. I wonder if anyone knows what can be gained in terms of separating out different topologies by allowing for a restricted family of conformal transformation on top of the tanh functions, of the sort that allows taking the insides out and so forth. Comments?
Re: Neural Networks, Manifolds, and Topology
#3Off topic: I wish, there would be a practical way to program for neuromorphic chips, which are inherently analog. Anybody have some idea or links regarding this field?
Re: Neural Networks, Manifolds, and Topology
#4This is very interesting. I wonder if anyone knows what can be gained in terms of separating out different topologies by allowing for a restricted family of conformal transformation on top of the tanh functions, of the sort that allows taking the insides out and so forth. Comments?
Why conformal? That seems a little too rigid.
Re: Neural Networks, Manifolds, and Topology
#5This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs.
In particular this:
> (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.)
Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but everybody does it anyway), but it could shed some light on the geometry of the "hard" instances.
EDIT: for example, to get a sense of how ill-conditioned is deep NN learning, a recent paper [1] shows that if we feed to an object classifying NN an image with crafted but unnoticeable perturbations the predictions change completely.
Re: Neural Networks, Manifolds, and Topology
#6A bit of an aside: Are there any methods for compressing/collapsing/simplifying a neural network.
What I mean is imagine you've built and trained a neural network, as per the article, it is hard to ascertain exactly what it is doing. I was wondering whether there is work in this area, and it occurred to me a possible first step would be to collapse the neural network to a simpler but functionally equivalent structure.
I imagine this is far more difficult than it sounds, but I just wondered.
Re: Neural Networks, Manifolds, and Topology
#7This is very interesting. I wonder if anyone knows what can be gained in terms of separating out different topologies by allowing for a restricted family of conformal transformation on top of the tanh functions, of the sort that allows taking the insides out and so forth. Comments?
Why conformal? That seems a little too rigid.
To get the effect I was referring to, yes it is too strict a class. I wasn't thinking very hard about it, but my idea was that conformal transformation have nice compositional properties, which could be exploited in developing the method and its analysis.
Re: Neural Networks, Manifolds, and Topology
#8Cascade correlation networks are known to be good at solving the spirals problem. They are also very fast to train.
Re: Neural Networks, Manifolds, and Topology
#9A bit of an aside: Are there any methods for compressing/collapsing/simplifying a neural network. What I mean is imagine you've built and trained a neural network, as per the article, it is hard to ascertain exactly what it is doing. I was wondering whether there is work in this area, and it occurred to me a possible first step would be to collapse the neural network to a simpler but functionally equivalent structure…
There are pruning methods which try to find and remove connections and neurons that don't have much effect on the network, essentially simplifying it.
There are also some methods such as HyperNeat which try to find simple structures that can be expanded into a much larger neural network. By sharing weights and having sections of the network that repeated many times.
Re: Neural Networks, Manifolds, and Topology
#10Bloody brilliant. I have longed for a ML methodology that breaks out of the float vector representation because I don't think it describes naturally occurring problems that well, most ontologies are semi-structured. However, this really gets to the nub of the problem, real, but solvable, problems are manifold complexes. I don't think anyone has really built a learner for complexes though, persistent homology is a stab in the right direction but its not there yet IMHO.