Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural Networks, Manifolds, and Topology (2014)

colah.github.io

Post reply on HN
Print view View original

1–10 of 26 posts

Re: Neural Networks, Manifolds, and Topology (2014)

#2

Post by flancian »

Previously:

https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114

But not a lot of discussion over there.

The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now.

  The manifold hypothesis is that natural data forms lower-dimensional
  manifolds in its embedding space. There are both theoretical and
  experimental reasons to believe this to be true. If you believe this, then
  the task of a classification algorithm is fundamentally to separate a bunch
  of tangled manifolds.
My interpretation/rephrasing: if you want to build a neural network that distinguishes cat and dog pictures, in the worst case that would seem to require a huge network with many nodes/layers (say, the number being a function of the size of the image) rather than the number that seems to work reasonably well in practice (six or some other rather low constant number observed in reality). So the number of dimensions over which the “images” are potentially spread is huge, but it’d seem that in the real world one can rearrange the dog and cat images in a “shape” that then allows for relatively easy disentanglement by the neural network; and these shapes can probably be realized in much lower dimensions (in the example, six).

This could explain (for some definition of explain) the observed predictive power of relatively small neural networks.

Re: Neural Networks, Manifolds, and Topology (2014)

#3

Post by LeanderK »

flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

well, the task of distinguishing cats and dogs is vastly different from the task of what defines a dog and a cat. You don't need to learn the underlying data-manifold, but just a decision-function between those two data-manifolds. We can't really build scalable models for the second problem yet, but have made great progress for the first one.

Re: Neural Networks, Manifolds, and Topology (2014)

#4

Post by BucketSort »

flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

If some data does form a manifold, then you just need to learn one continuous function per class. If it doesn't, then you need to learn several functions for different continuous regions per class. You need many more parameters to fit data that is fragmented essentially. "Separating tangled manifolds" is really learning the functions for each manifold. ( I may be off base here, the material is not fresh in my mind )

Re: Neural Networks, Manifolds, and Topology (2014)

#5

Post by scottlocklin »

flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting."

FWIIW unsupervised learning and stuff like topological data analysis is almost entirely about discovering the actual manifolds (or some hand wavey topology). Doesn't always work; the data often doesn't cooperate and live on a metric space.

Re: Neural Networks, Manifolds, and Topology (2014)

#9

Post by throwawaymath »

scottlocklin wrote: post #5
flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting." FWIIW unsupervised learning and stuff like topological data analysis is almost entirely about discovering the actual manifolds (or some hand wavey topology). Doesn't always work; the data often doesn't cooperate and live on a metric space.

That's a great way of saying it concisely. And to continue that generalization - one of the consequences of information theory is that there exist trivially incompressible strings. This plays nicely with the recent (2018) Lei-Luo-Yau-Gu result that there exist manifolds which cannot be learned.

As I harp on every chance I can get, I have a pet hypothesis that there's a very deep corollary here waiting to be proven rigorously. Namely, that we can show there exist adversarial inputs that exploit neural networks because they're incompressible. Furthermore, that these inputs are information theoretically guaranteed to exploit the neural network (even if there are practical complexity theoretic workarounds).

Post reply on HN