Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Applied Topology

theahura.substack.com

Post reply on HN
Print view View original

81–90 of 200 posts

Re: Deep Learning Is Applied Topology

#81

Post by theahura »

sota_pop wrote: post #4

I’ve always enjoyed this framing of the subject, the idea of mapping anything as hyperplanes existing in a solution space is one of the ideas that really blew my hair back during my academic studies. I would nitpick at your “dots in a circle example - with the stoner reference joke” I could be mistaken, but common practice isn’t to “move to a higher dimension”, but use a kernel (i.e. parameterize the points into the…

I'm pulling directly from Chris Olah's blog post with that example. But I will say that in practice, its always surprising how increasing the dimensionality of a neural network magically solves all sorts of problems. You could use a kernel if you don't have more computation available, but given more computation adding a dimension is strictly more flexible (and is capable of separating a much wider range of datasets)

Re: Deep Learning Is Applied Topology

#82

Post by fmap »

adamnemecek wrote: post #30
niemandhier wrote: post #18

Earlier quoted context omitted.

It’s alchemy. Deep learning in its current form relates to a hypothetical underlying theory as alchemy does to chemistry. In a few hundred years the Inuktitut speaking high schoolers of the civilisation that comes after us will learn that this strange word “deep learning” is a left over from the lingua franca of yore.

Not really, most of the current approaches are some approximations of the partition function.

The reason deep learning is alchemy is that none of these deep theories have predictive ability.

Essentially all practical models are discovered by trial and error and then "explained" after the fact. In many papers you read a few paragraphs of derivation followed by a simpler formulation that "works better in practice". E.g., diffusion models: here's how to invert the forward diffusion process, but actually we don't use this, because gradient descent on the inverse log likelihood works better. For bonus points the paper might come up with an impressive name for the simple thing.

In most other fields you would not get away with this. Your reviewers would point this out and you'd have to reformulate the paper as an experience report, perhaps with a section about "preliminary progress towards theoretical understanding". If your theory doesn't match what you do in practice - and indeed many random approaches will kind of work (!) - then it's not a good theory.

Re: Deep Learning Is Applied Topology

#83

Post by srean »

theahura wrote: post #75
srean wrote: post #29

The title, as it stands, is trite and wrong. More about that a little later. The article on the other hand is a pleasant read. Topology is whatever little structure that remains in geometry after you throwaway distances, angles, orientations and all sorts of non tearing stretchings. It's that bare minimum that still remains valid after such violent deformations. While notion of topology is definitely useful in machin…

> Topology is useful especially when you cannot trust lengths, distances angles and arbitrary deformations But...you can't. The input data lives on a manifold that you cannot 'trust'. It doesn't mean anything apriori that an image of a coca-cola can and an image of a stopsign live close to each other in pixel space. The neural network applies all of those violent transformations you are talking about

> But...you can't.

Only in a desperate sales pitch or a desparate research grants. There are of course some situations were certain measurements are untrustworthy, but to claim that is the common case is very snake oily.

When certain measurements become untrustworthy, that it does so only because of some unknown smooth transformation, is not very frequent (this is what purely topological methods will deal with). Random noise will also do that for you.

Not disputing the fact that sometimes metrics cannot be trusted entirely, but to go to a topological approach seems extreme. One should use as much of the relevant non-topological information as possible.

As the hackneyed example goes a topological methods would not be able to distinguish between a cup and a donut. For that you would need to trust non-topological features such as distances and angles. Deep learning methods can indeed differentiate between cop-nip and coffee mugs.

BTW I am completely on-board with the idea that data often looks as if it has been sampled from an unknown, potentially smooth, possibly non-Euclidean manifold and then corrupted by noise. In such cases recovering that manifold from noisy data is a very worthy cause.

In fact that is what most of your blogpost is about. But that's differential geometry and manifolds, they have structure far richer than a topology. For example they may have tangent planes, a Reimann metric or a symplectic form etc. A topological method would throw all of that away and focus on topology.

Re: Deep Learning Is Applied Topology

#84

Post by jebarker »

polotics wrote: post #5

Ok, how do transformers fit into this understanding of deep learning?

Transformers (with self-attention being the key operation) are kernel smoothers which fits easily into this view of the world. See here: http://bactra.org/notebooks/nn-attention-and-transformers.ht...

Re: Deep Learning Is Applied Topology

#85

Post by Graviscalar »

rubitxxx8 wrote: post #77
Graviscalar wrote: post #71

I was one of the people that was super excited after reading the Chris Olah blogpost from 2014, and over the past decade I've seen the insight go exactly nowhere. It's neat but it hasn't driven any interesting results, though Ayasdi did some interesting stuff with TDA and Gunnar Carlson has been playing around with neural nets recently.

What would you have expected to happen? Advances and insights sometimes lie dormant for decades or more before someone else picks them up and does something new.

I would expect model/algorithm improvements from using topological concepts to analyze the manifolds in question or concrete results in model interpretability. Gunnar has studied some toy examples, but they were barely a step up from the ones Olah constructed for the sake of explanation and they haven't borne any further fruit.

You can say any advance or insight is just lying dormant, it doesn't mean anything unless you can specifically articulate why it still has potential. I haven't made any claims on the future of the intersection of deep learning and topology, I was pointing out that it's been anything but dormant given the interest in it but it hasn't lead anywhere.

Re: Deep Learning Is Applied Topology

#86

Post by constantcrying »

umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

>it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

Neural Networks consist almost exclusively of two parts, numerical linear algebra and numerical optimization.

Even if you reject the abstract topological description. Numerical linear algebra and optimization couldn't be any more directly applicable.

Re: Deep Learning Is Applied Topology

#87

Post by adsharma »

Isn't Deep Learning more like Graph Theory? I shared yesterday that Google published a paper called CRISP (https://arxiv.org/pdf/2505.11471) that carefully avoids any reference to the word "Graph".

So then the question becomes what's the difference between Graph Theory and Applied Topology? Graphs operate on discrete structures and topology is about a continuous space. Otherwise they're very closely related.

But the higher order bit is that AI/ML and Deep Learning in particular could do a better job of learning from and acknowledging prior art from related fields. Reusing older terminology instead of inventing new.

Re: Deep Learning Is Applied Topology

#88

Post by wetpaws »

smsx wrote: post #63
almostgotcaught wrote: post #59

Earlier quoted context omitted.

You're citing a guy that never went to college (has no math or physics degree), has never published a paper, etc. I guess that actually tracks pretty well with how strong the whole "it's deep theory" claim is.

Chris Olah has never published a paper? ... https://scholar.google.com/citations?user=6dskOSUAAAAJ&hl=en...

[dead]

Re: Deep Learning Is Applied Topology

#89

Post by theahura »

fmap wrote: post #82
adamnemecek wrote: post #30

Earlier quoted context omitted.

Not really, most of the current approaches are some approximations of the partition function.

The reason deep learning is alchemy is that none of these deep theories have predictive ability. Essentially all practical models are discovered by trial and error and then "explained" after the fact. In many papers you read a few paragraphs of derivation followed by a simpler formulation that "works better in practice". E.g., diffusion models: here's how to invert the forward diffusion process, but actually we don't…

It's true that there is no directly predictive model of deep learning, and it's also true that there is some trial and error, but it is wrong to say that therefore there is no operating theory at all. I recommend reading Ilyas 30 papers (here's my review of that set: https://open.substack.com/pub/theahura/p/ilyas-30-papers-to-...) to see how shared intuitions and common threads are clearly developed over the last decade+

Re: Deep Learning Is Applied Topology

#90

Post by mitthrowaway2 »

srean wrote: post #51
ogogmad wrote: post #42

Earlier quoted context omitted.

The connections of deep learning to stat-mech and thermodynamics are really cool. It's led me to wonder about the origin of the probability distributions in stat-mech. Physical randomness is mostly a fiction (outside maybe quantum mechanics) so probability theory must be a convenient fiction. But objectively speaking, where then do the probabilities in stat-mech come from? So far, I've noticed that the (generalised)…

In Boltzmann's formulation of stat-mech it comes from the assumption that when a system is in "equilibrium", then all the micro-states that are consistent with the macro-state are equally occupied. That's the basis of the theory. A prime mover is thermal agitation. It can be circular if one defines equilibrium to be that situation when all the micro-states are equally occupied. One way out is to define equilibrium in…

The Bayesian reframing of that would be that when all you have measured is the macrostate, and you have no further information by which to assign a higher probability to any compatible microstate than any other, you follow the principle of indifference and assign a uniform distribution.
Post reply on HN