Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Applied Topology

theahura.substack.com

Post reply on HN
Print view View original

61–70 of 200 posts

Re: Deep Learning Is Applied Topology

#61

Post by danielmarkbruce »

thuuuomas wrote: post #8
umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

I cannot understand this prideful resentment of theory common among self-described practitioners. Even if existing theory is inadequate, would an operating theory not be beneficial? Or is the mystique combined with guess&check drudgery job security?

Who is proud? What you are seeing in some cases is eye rolling. And it's fair eye rolling.

There is an enormous amount of theory used in the various parts of building models, there just isn't an overarching theory at the very most convenient level of abstraction.

It almost has to be this way. If there was some neat theory, people would use it and build even more complex things on top of it in an experimental way and then so on.

Re: Deep Learning Is Applied Topology

#63

Post by smsx »

almostgotcaught wrote: post #59
theahura wrote: post #37

Earlier quoted context omitted.

I say this as someone who has been in deep learning for over a decade now: this is pretty wrong, both on the merits (data obviously lives on a manifold) and on its applications to deep learning (cf chris olah's blog as an example from 2014, which is linked in my post -- https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ). Embedding spaces are called 'spaces' for a reason. GANs, VAEs, contrastive losses --…

You're citing a guy that never went to college (has no math or physics degree), has never published a paper, etc. I guess that actually tracks pretty well with how strong the whole "it's deep theory" claim is.

Chris Olah has never published a paper? ... https://scholar.google.com/citations?user=6dskOSUAAAAJ&hl=en...

Re: Deep Learning Is Applied Topology

#64

Post by baxtr »

umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

Just a side comment to your observation: the principle is called reductionism and has been tried on many fields.

Physics is just applied mathematics

Chemistry is just applied physics

Biology is just applied chemistry

It doesn’t work very well.

Re: Deep Learning Is Applied Topology

#65

Post by fiddlerwoaroof »

canjobear wrote: post #39
thuuuomas wrote: post #8

Earlier quoted context omitted.

I cannot understand this prideful resentment of theory common among self-described practitioners. Even if existing theory is inadequate, would an operating theory not be beneficial? Or is the mystique combined with guess&check drudgery job security?

If there were theory that led to directly useful results (like, telling you the right hyperparameters to use for your data in a simple way, or giving you a new kind of regularization that you can drop in to dramatically improve learning) then deep learning practitioners would love it. As it currently stands, such theories don't really exist.

Useful theories only come to exist because someone started by saying they must exist and then spent years or lifetimes discovering them.

Re: Deep Learning Is Applied Topology

#66

Post by whatever1 »

kwertzzz wrote: post #28
sota_pop wrote: post #13

Earlier quoted context omitted.

I disagree with this wholeheartedly. Sure, there is lots of trial and error, but it’s more an amalgamation of theory from many areas of mathematics including but not limited to: topology, geometry, game theory, calculus, and statistics. The very foundations (i.e. back-propagation) is just the chain rule applied to the weights. The difference is that deep learning has become such an accessible (sic profitable) field t…

Can you give an example where theories and techniques from other fields are reinvented? I would be genuinely interested for concrete examples. Such "reinventions" happen quite often in science, so to some degree this would be expected.

I mean the entire domain of systems control is being reinvented by deep RL. System identification, stability, robustness etc

Re: Deep Learning Is Applied Topology

#67

Post by yubblegum »

umutisik wrote: post #3

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

> Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way.

Of course. Now, to actually deeply understand what is happening with these constructs, we will use topology. Topoligical insights will without doubt then inform the next generations of this technology.

Re: Deep Learning Is Applied Topology

#68

Post by theahura »

almostgotcaught wrote: post #59
theahura wrote: post #37

Earlier quoted context omitted.

I say this as someone who has been in deep learning for over a decade now: this is pretty wrong, both on the merits (data obviously lives on a manifold) and on its applications to deep learning (cf chris olah's blog as an example from 2014, which is linked in my post -- https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ). Embedding spaces are called 'spaces' for a reason. GANs, VAEs, contrastive losses --…

You're citing a guy that never went to college (has no math or physics degree), has never published a paper, etc. I guess that actually tracks pretty well with how strong the whole "it's deep theory" claim is.

Chris Olah? One of the founders of Anthropic and the head of their interpretability team?

Re: Deep Learning Is Applied Topology

#69

Post by qbit42 »

saberience wrote: post #50
sota_pop wrote: post #13

Earlier quoted context omitted.

I disagree with this wholeheartedly. Sure, there is lots of trial and error, but it’s more an amalgamation of theory from many areas of mathematics including but not limited to: topology, geometry, game theory, calculus, and statistics. The very foundations (i.e. back-propagation) is just the chain rule applied to the weights. The difference is that deep learning has become such an accessible (sic profitable) field t…

None of the major aspects of deep learning came from manifolds though. It is primarily linear algebra, calculus, probability theory and statistics, secondarily you could add something like information theory for ideas like entropy, loss functions etc. But really, if "manifolds" had never been invented/conceptualized, we would still have deep learning now, it really made zero impact on the actual practical technology…

Loss landscapes can be viewed as manifolds. Adagrad/ADAM adjust SGD to better fit the local geometry and are widely used in practice.

Re: Deep Learning Is Applied Topology

#70

Post by theahura »

canjobear wrote: post #39
thuuuomas wrote: post #8

Earlier quoted context omitted.

I cannot understand this prideful resentment of theory common among self-described practitioners. Even if existing theory is inadequate, would an operating theory not be beneficial? Or is the mystique combined with guess&check drudgery job security?

If there were theory that led to directly useful results (like, telling you the right hyperparameters to use for your data in a simple way, or giving you a new kind of regularization that you can drop in to dramatically improve learning) then deep learning practitioners would love it. As it currently stands, such theories don't really exist.

This is way too rigorous. You can absolutely have theories that lead to useful results even if they aren't as predictive as you describe. Theory of evolution for an obvious counterpoint.
Post reply on HN