Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Applied Topology

theahura.substack.com

Post reply on HN
Print view View original

41–50 of 200 posts

Re: Deep Learning Is Applied Topology

#41

Post by esafak »

umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

It does if you relax your definition to accommodate approximation error, cf. e.g., Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (https://aclanthology.org/2021.acl-long.568.pdf)

Re: Deep Learning Is Applied Topology

#42

Post by ogogmad »

srean wrote: post #34
kwertzzz wrote: post #28

Earlier quoted context omitted.

Can you give an example where theories and techniques from other fields are reinvented? I would be genuinely interested for concrete examples. Such "reinventions" happen quite often in science, so to some degree this would be expected.

Bethe ansatz is one. It took a toure de force by Yedidia to recognize that loopy belief propagation is computing the stationary point of Bethe's approximation to Free Energy. Many statistical thermodynamics ideas were reinvented in ML. Same is true for mirror descent. It was independently discovered by Warmuth and his students as Bregman divergence proximal minimization, or as a special case would have it, exponentia…

The connections of deep learning to stat-mech and thermodynamics are really cool.

It's led me to wonder about the origin of the probability distributions in stat-mech. Physical randomness is mostly a fiction (outside maybe quantum mechanics) so probability theory must be a convenient fiction. But objectively speaking, where then do the probabilities in stat-mech come from? So far, I've noticed that the (generalised) Boltzmann distribution serves as the bridge between probability theory and thermodynamics: It lets us take non-probabilistic physics and invent probabilities in a useful way.

Re: Deep Learning Is Applied Topology

#43

Post by [deleted] »

behnamoh wrote: post #12
umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

> a few intuitions coming from theory (that was not topology). I think these 'intuitions' are an after-the-fact thing, meaning AFTER deep learning comes up with a method, researchers in other fields of science notice the similarities between the deep learning approach and their (possibly decades old) methods. Here's an example where the author discovers that GPT is really the same computational problems he has solved…

[deleted]

Re: Deep Learning Is Applied Topology

#44

Post by csimon80 »

umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

"All models are wrong, but some are useful" -George Box

Re: Deep Learning Is Applied Topology

#45

Post by terabytest »

I'm confused by the author's diagram claiming that AGI/ASI are points on the same manifold as next token prediction, chat models, and CoT models. While the latter three are provably part of the same manifold, what justifies placing AGI/ASI there too?

What if the models capable of CoT aren't and will never be, regardless of topological manipulation, capable of processes that could be considered AGI? For example, human intelligence (the closest thing we know to AGI) requires extremely complex sensory and internal feedback loops and continuous processing unlike autoregressive models' discrete processing.

As a layman, this matches my intuition that LLMs are not at all in the same family of systems as the ones capable of generating intelligence or consciousness.

Re: Deep Learning Is Applied Topology

#46

Post by jvanderbot »

mjburgess wrote: post #35
soulofmischief wrote: post #26

Thanks for sharing. I also tend to view learning in terms of manifolds. It's a powerful representation. > I'm personally pretty convinced that, in a high enough dimensional space, this is indistinguishable from reasoning I actually have journaled extensively about this and even written some on Hacker News about it with respect to what I've been calling probabilistic reasoning manifolds: > This manifold is constructed…

Are you talking about reasoning in general, reasoning qua that mental process which operates on (representations of) propositions? In which case, I cannot understand " true reasoning is expressed in terms of probabilities, not axioms " One of the features of reasoning is that it does not operate in this way. It's highly implausible animals would have been endowed with no ability to operate non-probabilistically on pr…

I suspect, as a layperson who watches people make decisions all the time, that somewhere in our mind is a "certainty checker".

We don't do logic itself, we just create logic from certainty as part of verbal reasoning. It's our messy internal inference of likelihoods that causes us to pause and think, or dash forward with confidence, and convincing others is the only place we need things like "theorems".

This is the only way I can square things like intuition, writing to formalize thoughts, verbal argument, etc, with the fact that people are just so mushy all the time.

Re: Deep Learning Is Applied Topology

#47

Post by lostmsu »

profchemai wrote: post #22
esafak wrote: post #2

If it was topology we wouldn't bother to warp the manifold so we can do similarity search. No, it's geometry , with a metric. Just as in real life, we want to be able to compare things. Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? I imagine it violently changing at first before stabilizing, followed by geometric refinement.…

Agree, if anything it's Applied Linear Algebra...but that sounds less exotic.

Well, we know it is non-linear. More like differential equations.

Re: Deep Learning Is Applied Topology

#48

Post by lumost »

thuuuomas wrote: post #8
umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

I cannot understand this prideful resentment of theory common among self-described practitioners. Even if existing theory is inadequate, would an operating theory not be beneficial? Or is the mystique combined with guess&check drudgery job security?

There are many reasons to believe a theory may not be forthcoming, or that if it is available may not be useful.

For instance, we do not have consensus on what a theory should accomplish - should it provide convergence bounds/capability bounds? Should it predict optimal parameter counts/shapes? Should it allow more efficient calculation of optimal weights? Does it need to do these tasks in linear time?

Even materials science in metals is still cycling through theoretical models after thousands of years of making steel and other alloys.

Re: Deep Learning Is Applied Topology

#49

Post by theahura »

polotics wrote: post #5

Ok, how do transformers fit into this understanding of deep learning?

Transformers learn embedding representations of tokens, which are easily mapped into a space. Similar tokens are mapped to similar places on the space. The fully connected layer at the end of each transformer block defines a transformation of a set of points in a space to another point in that space, not unlike the example of adding colors together to get a new color

Re: Deep Learning Is Applied Topology

#50

Post by saberience »

sota_pop wrote: post #13
umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

I disagree with this wholeheartedly. Sure, there is lots of trial and error, but it’s more an amalgamation of theory from many areas of mathematics including but not limited to: topology, geometry, game theory, calculus, and statistics. The very foundations (i.e. back-propagation) is just the chain rule applied to the weights. The difference is that deep learning has become such an accessible (sic profitable) field t…

None of the major aspects of deep learning came from manifolds though.

It is primarily linear algebra, calculus, probability theory and statistics, secondarily you could add something like information theory for ideas like entropy, loss functions etc.

But really, if "manifolds" had never been invented/conceptualized, we would still have deep learning now, it really made zero impact on the actual practical technology we are all using every day now.

Post reply on HN