Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TopoNets: High performing vision and language models with brain-like topography

arxiv.org

Post reply on HN
Print view View original

61–70 of 76 posts

Re: TopoNets: High performing vision and language models with brain-like topography

#61

Post by photon_lines »

mayukhdeb wrote: post #53
brrrrrm wrote: post #50

Earlier quoted context omitted.

this paper plays into some popular fantasy about the aesthetic of ANNs. it’s not scientifically useful

If by popular fantasy you mean replicating the functional profiles of the visual and language cortex of the brain, then yes. These ideas in neuroscience are popular, but not fantasy. I encourage you to read up on functional organization in the brain, it's very fascinating. > it’s not scientifically useful Having structured weights in GPTs enables us to localize and control various concepts and study stuff like polyse…

I love the paper - don't read into the negative comments. I find that a lot of online feedback (more so on Reddit and much less so on HN (usually)) tends to be opinionated and misinformed by quite a bit these days. Fantastic work and fantastic read.

Re: TopoNets: High performing vision and language models with brain-like topography

#62

Post by exe34 »

energy123 wrote: post #5

The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. A…

> CNN will beat ViT on small data tasks, but that flips with enough scale because ViT imposes less inductive bias

any idea why this is the case? CNN have the bias that neighbouring pixels are somehow relevant - they are neighbours. ViTs have to re-learn this from scratch. So why do they end up doing better than CNN?

Re: TopoNets: High performing vision and language models with brain-like topography

#63

Post by chaumaha »

light_hue_1 wrote: post #9

They bury the part where inducing brain like structure hurts performance! This is a method to just hurt your network in exchange for nothing useful at all aside from some sketchy story that this is "brain like".

[dead]

Re: TopoNets: High performing vision and language models with brain-like topography

#64

Post by chaumaha »

energy123 wrote: post #5

The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. A…

[dead]

Re: TopoNets: High performing vision and language models with brain-like topography

#65

Post by chaumaha »

gzer0 wrote: post #39

I spent time working with Andrej and the rest of the FSD team back in 2020/2021, and we had plenty of conversations on how human visual processing maps onto our neural network architectures. Our approach—transformer-based attention blocks, multi-scale feature extraction, and temporal fusion—mirrors elements of the biological visual cortex (retina → LGN → V1 → V2 → V4 → IT) which break down raw inputs and integrate th…

[dead]

Re: TopoNets: High performing vision and language models with brain-like topography

#66

Post by mayukhdeb »

photon_lines wrote: post #61
mayukhdeb wrote: post #53

Earlier quoted context omitted.

If by popular fantasy you mean replicating the functional profiles of the visual and language cortex of the brain, then yes. These ideas in neuroscience are popular, but not fantasy. I encourage you to read up on functional organization in the brain, it's very fascinating. > it’s not scientifically useful Having structured weights in GPTs enables us to localize and control various concepts and study stuff like polyse…

I love the paper - don't read into the negative comments. I find that a lot of online feedback (more so on Reddit and much less so on HN (usually)) tends to be opinionated and misinformed by quite a bit these days. Fantastic work and fantastic read.

Thank you for your kind words!

Re: TopoNets: High performing vision and language models with brain-like topography

#67

Post by jlpom »

energy123 wrote: post #5

The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. A…

It increases modularity and small-worldness, which are in my book critical for AGI (surprised by the way that this publication doesn't cite https://www.nature.com/articles/s42256-023-00748-9).

Re: TopoNets: High performing vision and language models with brain-like topography

#68

Post by mayukhdeb »

jlpom wrote: post #67
energy123 wrote: post #5

The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. A…

It increases modularity and small-worldness, which are in my book critical for AGI (surprised by the way that this publication doesn't cite https://www.nature.com/articles/s42256-023-00748-9 ).

Thank you for sharing this! We'll read through this and update the camera-ready version accordingly for ICLR 2025.

Re: TopoNets: High performing vision and language models with brain-like topography

#69

Post by brrrrrm »

mayukhdeb wrote: post #53
brrrrrm wrote: post #50

Earlier quoted context omitted.

this paper plays into some popular fantasy about the aesthetic of ANNs. it’s not scientifically useful

If by popular fantasy you mean replicating the functional profiles of the visual and language cortex of the brain, then yes. These ideas in neuroscience are popular, but not fantasy. I encourage you to read up on functional organization in the brain, it's very fascinating. > it’s not scientifically useful Having structured weights in GPTs enables us to localize and control various concepts and study stuff like polyse…

I probably came in too hot on that (dealing with some personal stuff). Although I disagree with purported the impact of the paper, I don’t think this is fundamentally incorrect or bad science and I wish you the best on future research.

Re: TopoNets: High performing vision and language models with brain-like topography

#70

Post by devmor »

mayukhdeb wrote: post #51
devmor wrote: post #35

Is this "brain-like" in any functional way, or "brain-like" in the same way that a tall rectangle is "door-like" even if it doesn't share any functions with a door? I know quite a bit about machine learning, but very little to nothing about neuroscience and human cognition, so I am curious how an expert (that didn't work on the paper) would describe it. (Forgive me for the pre-emptive negativity but I am so utterly e…

It is indeed brain-like in a functional way. Topographic structure is what enables the brain to have low dimensionality and metabolic efficiency. We find that inducing such structure in neural nets made them have significantly lower dimensionality and also more parameter efficient (After training, we could take advantage of the structure to remove ~80% of the weights in topographic layers without sacrificing performa…

>After training, we could take advantage of the structure to remove ~80% of the weights in topographic layers without sacrificing performance

This is really interesting to me. Is it that the structure clustered the neurons in such a way that they didn't need to be weighted because their function were grouped by similar black box properties?

Post reply on HN