Neural Networks, Manifolds, and Topology (2014) • Hacker News Board

Re: Neural Networks, Manifolds, and Topology (2014)

#21

Post by srean » Thu, Feb 14, 2019, 4:32 PM UTC

scottlocklin wrote: post #5

flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting." FWIIW unsupervised learning and stuff like topological data analysis is almost entirely about discovering the actual manifolds (or some hand wavey topology). Doesn't always work; the data often doesn't cooperate and live on a metric space.

"compression is the same thing as forecasting"

This is true by definition regardless of manifold hypothesis.

In order to define compression you dont need a nontrivial metric or a topological space. You need things to even talk about the manifold hypothesis, at least in any interesting way.

Re: Neural Networks, Manifolds, and Topology (2014)

#22

Post by srean » Thu, Feb 14, 2019, 4:38 PM UTC

cafebeen wrote: post #13

throwawaymath wrote: post #12

Earlier quoted context omitted.

> To play devil's advocate... given that manifolds are so general, wouldn't it be more surprising if natural data didn't form some lower dimensional manifold? No, because not all data can actually be represented on a topology. It's nice if you can model your data as a normed vector space, but it's not inherently possible in general.

In a trivial sense, it seems that a Euclidean topology would do the trick, no? I'd be curious to hear counter-examples, of course. In my mind, I suppose what's missing from the "manifold hypothesis", as stated above, is that the manifolds should be more useful than raw data, for example, does their structure respect our notion of object categories or are they sufficiently low-dimensional for visualization?

They key is the low dimensional part not that it is a manifold.

The ambient space may be huge, but data is not spread all over but lies on a tiny subset that can be well described by very few parameters. This limited degree of freedom in the data is what makes it easy to learn. A priori there is no reason why data should have such a property.

I would be interested in your thoughts on how you would map human language understanding to a Euclidean topology.

Re: Neural Networks, Manifolds, and Topology (2014)

#23

Post by scottlocklin » Thu, Feb 14, 2019, 6:01 PM UTC

srean wrote: post #21

scottlocklin wrote: post #5

Earlier quoted context omitted.

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting." FWIIW unsupervised learning and stuff like topological data analysis is almost entirely about discovering the actual manifolds (or some hand wavey topology). Doesn't always work; the data often doesn't cooperate and live on a metric space.

"compression is the same thing as forecasting" This is true by definition regardless of manifold hypothesis. In order to define compression you dont need a nontrivial metric or a topological space. You need things to even talk about the manifold hypothesis, at least in any interesting way.

Well I'll say it a different way: the manifold hypothesis is basically a geometric restatement of the fact that compression is the same thing as forecasting.

If you couldn't map a metric space (or other distance measure) onto a lower dimensional manifold, you're basically saying you can't forecast using a geometric representation, or you have already extracted all the signal from the noise in some other way.

There are interesting problems where you can't do anything with metrics or quasi-metric distance measures, but you can forecast things. Geometry is easier to reason about than Kolmogorov complexity though, hence papers like the above.

Re: Neural Networks, Manifolds, and Topology (2014)

#24

Post by srean » Thu, Feb 14, 2019, 6:40 PM UTC

scottlocklin wrote: post #23

srean wrote: post #21

Earlier quoted context omitted.

"compression is the same thing as forecasting" This is true by definition regardless of manifold hypothesis. In order to define compression you dont need a nontrivial metric or a topological space. You need things to even talk about the manifold hypothesis, at least in any interesting way.

Well I'll say it a different way: the manifold hypothesis is basically a geometric restatement of the fact that compression is the same thing as forecasting. If you couldn't map a metric space (or other distance measure) onto a lower dimensional manifold, you're basically saying you can't forecast using a geometric representation, or you have already extracted all the signal from the noise in some other way. There ar…

Not quite buying that yet. Why would one need a metric ?

Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology.

One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?

Re: Neural Networks, Manifolds, and Topology (2014)

#25

Post by scottlocklin » Thu, Feb 14, 2019, 6:53 PM UTC

srean wrote: post #24

scottlocklin wrote: post #23

Earlier quoted context omitted.

Well I'll say it a different way: the manifold hypothesis is basically a geometric restatement of the fact that compression is the same thing as forecasting. If you couldn't map a metric space (or other distance measure) onto a lower dimensional manifold, you're basically saying you can't forecast using a geometric representation, or you have already extracted all the signal from the noise in some other way. There ar…

Not quite buying that yet. Why would one need a metric ? Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology. One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?

Hey man, I didn't post the paper on neural nets, manifolds and topology; I'm not going to defend it.

I already said you don't need a metric. But in the presence of an interesting metric or other distance measure; of course the signal fits on a lower dimensional manifold. Metric spaces and other distance measures give people a lot of tools to reason about data in general, hence virtually all common unsupervised learning algorithms and stuff like topological data analysis.

Re: Neural Networks, Manifolds, and Topology (2014)

#26

Post by srean » Thu, Feb 14, 2019, 7:31 PM UTC

scottlocklin wrote: post #25

srean wrote: post #24

Earlier quoted context omitted.

Not quite buying that yet. Why would one need a metric ? Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology. One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?

Hey man, I didn't post the paper on neural nets, manifolds and topology; I'm not going to defend it. I already said you don't need a metric. But in the presence of an interesting metric or other distance measure; of course the signal fits on a lower dimensional manifold. Metric spaces and other distance measures give people a lot of tools to reason about data in general, hence virtually all common unsupervised learni…

Oh the post is a good one. What I was drawing attention to is that manifold hypothesis and the compression hypothesis are different concepts. The compression hypothesis is more general to the point of being a definition. It holds true regardless of whether the low dimensional manifold property holds or not.