Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural Networks, Manifolds, and Topology (2014)

colah.github.io

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Neural Networks, Manifolds, and Topology (2014)

#11

Post by cafebeen »

flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

To play devil's advocate... given that manifolds are so general, wouldn't it be more surprising if natural data didn't form some lower dimensional manifold? That is, a manifold only requires some consistent notion of neighborhoods and a way to locally parameterize objects, which is a fairly low bar to pass, and perhaps trivial if you knock off one pixel in the case of images. Maybe the surprising point (which is not said explicitly) is that the manifold is very low dimensional compared to sensor data, but I suppose it's hard to formulate a hypothesis about the magnitude of the reduction besides it being "lower".

Re: Neural Networks, Manifolds, and Topology (2014)

#12

Post by throwawaymath »

cafebeen wrote: post #11
flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

To play devil's advocate... given that manifolds are so general, wouldn't it be more surprising if natural data didn't form some lower dimensional manifold? That is, a manifold only requires some consistent notion of neighborhoods and a way to locally parameterize objects, which is a fairly low bar to pass, and perhaps trivial if you knock off one pixel in the case of images. Maybe the surprising point (which is not…

> To play devil's advocate... given that manifolds are so general, wouldn't it be more surprising if natural data didn't form some lower dimensional manifold?

No, because not all data can actually be represented on a topology. It's nice if you can model your data as a normed vector space, but it's not inherently possible in general.

Re: Neural Networks, Manifolds, and Topology (2014)

#13

Post by cafebeen »

throwawaymath wrote: post #12
cafebeen wrote: post #11

Earlier quoted context omitted.

To play devil's advocate... given that manifolds are so general, wouldn't it be more surprising if natural data didn't form some lower dimensional manifold? That is, a manifold only requires some consistent notion of neighborhoods and a way to locally parameterize objects, which is a fairly low bar to pass, and perhaps trivial if you knock off one pixel in the case of images. Maybe the surprising point (which is not…

> To play devil's advocate... given that manifolds are so general, wouldn't it be more surprising if natural data didn't form some lower dimensional manifold? No, because not all data can actually be represented on a topology. It's nice if you can model your data as a normed vector space, but it's not inherently possible in general.

In a trivial sense, it seems that a Euclidean topology would do the trick, no? I'd be curious to hear counter-examples, of course. In my mind, I suppose what's missing from the "manifold hypothesis", as stated above, is that the manifolds should be more useful than raw data, for example, does their structure respect our notion of object categories or are they sufficiently low-dimensional for visualization?

Re: Neural Networks, Manifolds, and Topology (2014)

#14

Post by angel_j »

scottlocklin wrote: post #5
flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting." FWIIW unsupervised learning and stuff like topological data analysis is almost entirely about discovering the actual manifolds (or some hand wavey topology). Doesn't always work; the data often doesn't cooperate and live on a metric space.

That's if you go about it from low-dimensional embedding to feature classification, and predictions based on probabilities for those features. As you mention, that works if you have metric data, or can juice metrics from abstraction. Modern ML is all about solving for this case; adding manifold hypothesis, as the author noted, may only improve them slightly, or not at all.

That isn't what the "manifold hypothesis" is good for. Manifold space is more interesting to think of, not in the deconstruction of metric space, but in the formation of it; i.e. what higher-dimensional, non-metric (intensive), manifold-space, through some complex function (eg physics), results in this lower-dimensional, feature-rich entity; and what is that function (or more interestingly, how does it change the resulting entity-state, given various changes in the manifolds).

Which may not be useful for classification, but that's old hat anyway. Personally, I feel that feature dependence for everything NN is brute force. How features come-to-be, even abstractly modeled, is the thing.

Re: Neural Networks, Manifolds, and Topology (2014)

#15

Post by jvkersch »

cafebeen wrote: post #11
flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

To play devil's advocate... given that manifolds are so general, wouldn't it be more surprising if natural data didn't form some lower dimensional manifold? That is, a manifold only requires some consistent notion of neighborhoods and a way to locally parameterize objects, which is a fairly low bar to pass, and perhaps trivial if you knock off one pixel in the case of images. Maybe the surprising point (which is not…

A counterexample would be a dataset that forms a fractal in the ambient space. I don't know of a realistic example of this, but it seems plausible if you think of scale-invariant phenomena. Other ways of getting fractal-like structures, or at least "dust-like" structures with weird topologies, is by taking intersections of smooth structures. These things would be hard to separate...

Re: Neural Networks, Manifolds, and Topology (2014)

#16

Post by yantrams »

This was the article that helped me get neural networks when I began studying them few years back. Interpreting them as a series of curvillinear coordinate transformations really helped me understand them better.

PS: There is a great introductory article on entropy on the blog that is worth checking out.

Re: Neural Networks, Manifolds, and Topology (2014)

#17

Post by whatshisface »

throwawaymath wrote: post #9
scottlocklin wrote: post #5

Earlier quoted context omitted.

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting." FWIIW unsupervised learning and stuff like topological data analysis is almost entirely about discovering the actual manifolds (or some hand wavey topology). Doesn't always work; the data often doesn't cooperate and live on a metric space.

That's a great way of saying it concisely. And to continue that generalization - one of the consequences of information theory is that there exist trivially incompressible strings. This plays nicely with the recent (2018) Lei-Luo-Yau-Gu result that there exist manifolds which cannot be learned. As I harp on every chance I can get, I have a pet hypothesis that there's a very deep corollary here waiting to be proven ri…

>Furthermore, that these inputs are information theoretically guaranteed to exploit the neural network (even if there are practical complexity theoretic workarounds).

I get the image of a technique that, when applied to humans, allows you to see through political speeches and reveals the eldritch horrors scurrying around us continually.

Re: Neural Networks, Manifolds, and Topology (2014)

#19

Post by max_ »

flancian wrote: post #2

Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to belie…

How do people even submit URLs more than once on Hacker News?

Re: Neural Networks, Manifolds, and Topology (2014)

#20

Post by brookhaven_dude »

GlenTheMachine wrote: post #7

If anyone could point me to literature on k-nn neural networks (or the relationship, if any, between k-nn algorithms and basis function decomposition and/or blind source separation) I’d be much obliged.

He mentioned that he applied knn to handwritten digit recognition. A likely approach would be as follows.

Given a new vector that needs to be classified (say, x), it is compared with its nearest neighbors in the data set (let's call them x_1, x_2, x_3,..,x_k). A weighted average of the categories of the nearest neighbors is calculated to classify x.

That is to say, the category y that x belongs to is given by some function of categories of the nearest neighbor (e.g. weighted sum)

y = f(w_1y_1 + .... + w_ky_k)

where f() is a function that converts the continuous weighted sum to one of the integers representing the categories, and y_1,...,y_k are categories that x_1,...,x_k belong to, respectively.

The weights w_1,....,w_k can be determined by optimizing some error function.

Post reply on HN