I was one of the people that was super excited after reading the Chris Olah blogpost from 2014, and over the past decade I've seen the insight go exactly nowhere. It's neat but it hasn't driven any interesting results, though Ayasdi did some interesting stuff with TDA and Gunnar Carlson has been playing around with neural nets recently.
Deep Learning Is Applied Topology
71–80 of 200 posts
Re: Deep Learning Is Applied Topology
#72Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).
> Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Of course. Now, to actually deeply understand what is happening with these constructs, we will use topology. Topoligical insights will without doubt then inform the next generations of this technology.
May I ask you to give examples of insights from topology which improved existing models, or at least improved our understanding of them? arxiv papers are preferred.
Re: Deep Learning Is Applied Topology
#73If it was topology we wouldn't bother to warp the manifold so we can do similarity search. No, it's geometry , with a metric. Just as in real life, we want to be able to compare things. Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? I imagine it violently changing at first before stabilizing, followed by geometric refinement.…
> Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training?
If you've ever played with GANs or VAEs, you can actually answer this question! And the answer is more or less 'yes'. You can look at GANs at various checkpoints during training and see how different points in the high dimensional space move around (using tools like UMAP / TSNE).
> I imagine it violently changing at first before stabilizing, followed by geometric refinement
Also correct, though the violent changing at the beginning is also influenced the learning rate and the choice of optimizer.
Re: Deep Learning Is Applied Topology
#74Earlier quoted context omitted.
> Data doesn't actually live on a manifold. Often, they do (and then they are called "sheaves").
Many types of data don’t. Disconnected spaces like integer spaces don’t sit on a manifold (they are lattices). Spiky noisy fragmented data don’t sit on a (smooth) manifold. In fact not all ML models treat data as manifolds. Nearest neighbors, decision trees don’t require the manifold assumption and actually work better without it.
Any reasonable statistical explanation of deep learning requires there to be some sort of low dimensional latent structure in the data. Otherwise, we would not have enough training data to learn good models, given how high the ambient dimensions are for most problems.
Re: Deep Learning Is Applied Topology
#75The title, as it stands, is trite and wrong. More about that a little later. The article on the other hand is a pleasant read. Topology is whatever little structure that remains in geometry after you throwaway distances, angles, orientations and all sorts of non tearing stretchings. It's that bare minimum that still remains valid after such violent deformations. While notion of topology is definitely useful in machin…
> Topology is useful especially when you cannot trust lengths, distances angles and arbitrary deformations
But...you can't. The input data lives on a manifold that you cannot 'trust'. It doesn't mean anything apriori that an image of a coca-cola can and an image of a stopsign live close to each other in pixel space. The neural network applies all of those violent transformations you are talking about
Re: Deep Learning Is Applied Topology
#76If it was topology we wouldn't bother to warp the manifold so we can do similarity search. No, it's geometry , with a metric. Just as in real life, we want to be able to compare things. Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? I imagine it violently changing at first before stabilizing, followed by geometric refinement.…
> Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? If you've ever played with GANs or VAEs, you can actually answer this question! And the answer is more or less 'yes'. You can look at GANs at various checkpoints during training and see how different points in the high dimensional space move around (using tools like UMAP / TSNE…
And crucially, the initialization algorithm.
Re: Deep Learning Is Applied Topology
#77I was one of the people that was super excited after reading the Chris Olah blogpost from 2014, and over the past decade I've seen the insight go exactly nowhere. It's neat but it hasn't driven any interesting results, though Ayasdi did some interesting stuff with TDA and Gunnar Carlson has been playing around with neural nets recently.
What would you have expected to happen?
Advances and insights sometimes lie dormant for decades or more before someone else picks them up and does something new.
Re: Deep Learning Is Applied Topology
#78I really liked this article, though I don't know why the author is calling the idea of finding a separating surface between two classes of points "topology." For instance, they write "If you are trying to learn a translation task — say, English to Spanish, or Images to Text — your model will learn a topology where bread is close to pan, or where that picture of a cat is close to the word cat." This is everything that…
fair, I was using 'topology' more colloquially in that sentence. Should have said 'surface'.
Re: Deep Learning Is Applied Topology
#79[flagged]
Re: Deep Learning Is Applied Topology
#80I'm confused by the author's diagram claiming that AGI/ASI are points on the same manifold as next token prediction, chat models, and CoT models. While the latter three are provably part of the same manifold, what justifies placing AGI/ASI there too? What if the models capable of CoT aren't and will never be, regardless of topological manipulation, capable of processes that could be considered AGI? For example, human…
Possible. AGI/ASI are poorly defined. I tend to think we're already at AGI, obviously many disagree.
> For example, human intelligence (the closest thing we know to AGI) requires extremely complex sensory and internal feedback loops and continuous processing unlike autoregressive models' discrete processing.
I've done a fair bit of connectomics research and I think that this framing elides the ways in which neural networks and biological networks are actually quite similar. For example, in mice olfactory systems there is something akin to a 'feature vector' that appears based on which neurons light up. Specific sets of neurons lighting up means 'chocolate' or 'lemon' or whatever. More generally, it seems like neuronal representations are somewhat similar to embedding representations, and you could imagine constructing an embedding space based on what neurons light up where. Everything on top of the embeddings is 'just' processing.