Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Applied Topology

theahura.substack.com

Post reply on HN
Print view View original

151–160 of 200 posts

Re: Deep Learning Is Applied Topology

#151

Post by quantadev »

srean wrote: post #147
quantadev wrote: post #144

Earlier quoted context omitted.

The most common uses of "topology", whenever used to convey a geometry-related idea, is in the more general sense meaning "surfaces". Only one out of a million times is anyone ever referring to the specific mathematical field of the same name to which you refer.

LOL the very first line of your own link (that now you seem to have deleted after my comment https://en.m.wikipedia.org/wiki/Topology_(disambiguation) ) "Topology is a branch of mathematics concerned with geometric properties preserved under continuous deformation (stretching without tearing or gluing)" That is indeed the established meaning of topology, more so in mathematics and the blog post was on applied mathema…

The question was never "Is topology a field of mathematics". The question was, is that term most often used to refer to surfaces in general, and the answer to that is still 'yes'.

Re: Deep Learning Is Applied Topology

#152

Post by dang »

colah3 wrote: post #98

Since this post is based on my 2014 blog post ( https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment. I tried really hard to use topology as a way to understand neural networks, for example in these follow ups: - https://colah.github.io/posts/2014-10-Visualizing-MNIST/ - https://colah.github.io/posts/2015-01-Visualizing-Representa... There are places I've found the topological per…

That earlier post had a few small HN discussions (for those interested):

Neural Networks, Manifolds, and Topology (2014) - https://news.ycombinator.com/item?id=19132702 - Feb 2019 (25 comments)

Neural Networks, Manifolds, and Topology (2014) - https://news.ycombinator.com/item?id=9814114 - July 2015 (7 comments)

Neural Networks, Manifolds, and Topology - https://news.ycombinator.com/item?id=7557964 - April 2014 (29 comments)

Re: Deep Learning Is Applied Topology

#153

Post by srean »

quantadev wrote: post #151
srean wrote: post #147

Earlier quoted context omitted.

LOL the very first line of your own link (that now you seem to have deleted after my comment https://en.m.wikipedia.org/wiki/Topology_(disambiguation) ) "Topology is a branch of mathematics concerned with geometric properties preserved under continuous deformation (stretching without tearing or gluing)" That is indeed the established meaning of topology, more so in mathematics and the blog post was on applied mathema…

The question was never "Is topology a field of mathematics". The question was, is that term most often used to refer to surfaces in general, and the answer to that is still 'yes'.

https://www.amazon.com/s?k=Topology&sprefix=topology+%2Caps%...

Could you drop Amazon a message. They should really change the search results if that's what topology means in general. I am sure they would be delighted to receive the bug report.

For laughs, I asked ChatGPT to use 'topology' in place of 'surface'. Here's what it wrote:

    The topology of the lake was so calm it reflected the mountains perfectly.

    She wiped the kitchen topology clean after cooking dinner.

    After years of silence, the truth began to topology.

    The spacecraft landed safely on the topology of Mars.

    A thin layer of dust had settled on the topology of the old table.

    He barely scratched the topology of the topic in his presentation.

    As the submarine ascended, it broke through the topology of the ocean.

    The topology of the road was slick with ice.

    Despite her calm topology, she was extremely nervous inside.

    The paint bubbled and peeled off the topology due to the heat.
Our disagreement aside, I think these are hilarious. We should agree on that.

The best was

    The surface of a doughnut resembles that of a coffee mug due to their similar structure.

Re: Deep Learning Is Applied Topology

#154

Post by jxramos »

Graviscalar wrote: post #71

I was one of the people that was super excited after reading the Chris Olah blogpost from 2014, and over the past decade I've seen the insight go exactly nowhere. It's neat but it hasn't driven any interesting results, though Ayasdi did some interesting stuff with TDA and Gunnar Carlson has been playing around with neural nets recently.

Ayasdi immediately came to mind too seeing this post. I haven't thought of them in a long time, looks like they got bought out in 2019, prepandemic too which was probably best since mid pandemic had a lot of poor valuations

https://www.symphonyai.com/news/financial-services/ayasdi-jo...

Re: Deep Learning Is Applied Topology

#155

Post by [deleted] »

srean wrote: post #153
quantadev wrote: post #151

Earlier quoted context omitted.

The question was never "Is topology a field of mathematics". The question was, is that term most often used to refer to surfaces in general, and the answer to that is still 'yes'.

https://www.amazon.com/s?k=Topology&sprefix=topology+%2Caps%... Could you drop Amazon a message. They should really change the search results if that's what topology means in general. I am sure they would be delighted to receive the bug report. For laughs, I asked ChatGPT to use 'topology' in place of 'surface'. Here's what it wrote: The topology of the lake was so calm it reflected the mountains perfectly. She wiped…

[deleted]

Re: Deep Learning Is Applied Topology

#156

Post by colah3 »

CliffStoll wrote: post #136

Applied topology. Might a Klein bottle actually be a useful?

It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equivariance/#hue-rotation...

(This is, in some sense, for similar reason to Gunnar Carlson et al finding a Klein bottle when looking at high-contrast image patches, except one level more abstract, since it's about features rather than data points.)

Re: Deep Learning Is Applied Topology

#157

Post by TimorousBestie »

srean wrote: post #147
quantadev wrote: post #144

Earlier quoted context omitted.

The most common uses of "topology", whenever used to convey a geometry-related idea, is in the more general sense meaning "surfaces". Only one out of a million times is anyone ever referring to the specific mathematical field of the same name to which you refer.

LOL the very first line of your own link (that now you seem to have deleted after my comment https://en.m.wikipedia.org/wiki/Topology_(disambiguation) ) "Topology is a branch of mathematics concerned with geometric properties preserved under continuous deformation (stretching without tearing or gluing)" That is indeed the established meaning of topology, more so in mathematics and the blog post was on applied mathema…

> Alternatively, I would say, take a breath. Is this hill really the one worth dying on ?

This applies both ways. Maybe you could relax with the facetiousness? It doesn’t help your argument and makes you look like an ass.

Re: Deep Learning Is Applied Topology

#158

Post by naasking »

mjburgess wrote: post #129
naasking wrote: post #124

Earlier quoted context omitted.

> It's highly implausible animals would have been endowed with no ability to operate non-probabilistically on propositions represented by them, since this is essential for correct reasoning Why would animals need to evolve 100% correct reasoning if probabilistically correct reasoning suffices? If probabilistic reasoning is cheaper in terms of energy then correct reasoning is a disadvantage.

It doesnt suffice. It's also vastly energetically cheaper just to have (algorithmic) negation. Compressing (A, not A) into a probability function is extremely incomprehensibly expensive.

> It's also vastly energetically cheaper just to have (algorithmic) negation.

Even if true, that's an argument that it's cheaper to have something, not that it's cheaper to develop it through natural selection. Training time and energy for LLMs shows how energy intensive training to get to the point of grokking/circuit generalization.

Re: Deep Learning Is Applied Topology

#159

Post by HarHarVeryFunny »

colah3 wrote: post #142
montebicyclelo wrote: post #108

Earlier quoted context omitted.

Related to ways of understanding neural networks, I've seen these views expressed a lot, which to me seem like misconceptions: - LLMs are basically just slightly better `n-gram` models - The idea of "just" predicting the next token, as if next-token-prediction implies a model must be dumb (I wonder if this [1] popular response to Karpathy's RNN [2] post is partly to blame for people equating language neural nets with…

I guess I'll plug my hobby horse: The whole discourse of "stochastic parrots" and "do models understand" and so on is deeply unhealthy because it should be scientific questions about mechanism, and people don't have a vocabulary for discussing the range of mechanisms which might exist inside a neural network. So instead we have lots of arguments where people project meaning onto very fuzzy ideas and the argument does…

1) Isn't it unavoidable that a transformer - a sequential multi-layer architecture - is doing multi-step inference ?!

2) There are two aspects to a rhyming poem:

a) It is a poem, so must have a fairly high degree of thematic coherence

b) It rhymes, so must have end-of-line rhyming words

It seems that to learn to predict (hence generate) a rhyming poem, both of these requirements (theme/story continuation+rhyming) would need to be predicted ("planned") at least by the beginning of the line, since they are inter-related.

In contrast, a genre like freestyle rap may also rhyme, but flow is what matters and thematic coherence and rhyming may suffer as a result. In learning to predict (hence generate) freestyle, an LLM might therefore be expected to learn that genre-specific improv is what to expect, and that rhyming is of secondary importance, so one might expect less rhyme-based prediction ("planning") at the start of each bar (line).

Re: Deep Learning Is Applied Topology

#160

Post by [deleted] »

TimorousBestie wrote: post #157
srean wrote: post #147

Earlier quoted context omitted.

LOL the very first line of your own link (that now you seem to have deleted after my comment https://en.m.wikipedia.org/wiki/Topology_(disambiguation) ) "Topology is a branch of mathematics concerned with geometric properties preserved under continuous deformation (stretching without tearing or gluing)" That is indeed the established meaning of topology, more so in mathematics and the blog post was on applied mathema…

> Alternatively, I would say, take a breath. Is this hill really the one worth dying on ? This applies both ways. Maybe you could relax with the facetiousness? It doesn’t help your argument and makes you look like an ass.

[deleted]
Post reply on HN