Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TopoNets: High performing vision and language models with brain-like topography

arxiv.org

Post reply on HN
Print view View original

71–76 of 76 posts

Re: TopoNets: High performing vision and language models with brain-like topography

#71

Post by mayukhdeb »

devmor wrote: post #70
mayukhdeb wrote: post #51

Earlier quoted context omitted.

It is indeed brain-like in a functional way. Topographic structure is what enables the brain to have low dimensionality and metabolic efficiency. We find that inducing such structure in neural nets made them have significantly lower dimensionality and also more parameter efficient (After training, we could take advantage of the structure to remove ~80% of the weights in topographic layers without sacrificing performa…

>After training, we could take advantage of the structure to remove ~80% of the weights in topographic layers without sacrificing performance This is really interesting to me. Is it that the structure clustered the neurons in such a way that they didn't need to be weighted because their function were grouped by similar black box properties?

> Is it that the structure clustered the neurons in such a way that they didn't need to be weighted

Yep. Because of the structure, we did not have to compute the output of each weight column and simply copied the outputs of nearby weight columns whose outputs were computed.

Re: TopoNets: High performing vision and language models with brain-like topography

#72

Post by mayukhdeb »

vlovich123 wrote: post #60
harles wrote: post #18

Earlier quoted context omitted.

That could explain compute efficiency, but has nothing to do with the parameter efficiency pointed at in the paper.

Haven’t read the paper but my guess around that is that the same reason sparse attention networks (where they 0 out many weights) just have the sparse tensors be larger.

In this paper, we don't zero out the weights. We remove them.

Re: TopoNets: High performing vision and language models with brain-like topography

#73

Post by vlovich123 »

mayukhdeb wrote: post #72
vlovich123 wrote: post #60

Earlier quoted context omitted.

Haven’t read the paper but my guess around that is that the same reason sparse attention networks (where they 0 out many weights) just have the sparse tensors be larger.

In this paper, we don't zero out the weights. We remove them.

Thanks for the correction! Can it be retrofitted into existing models through distillation or do you have to train the model from scratch?

Re: TopoNets: High performing vision and language models with brain-like topography

#74

Post by devmor »

mayukhdeb wrote: post #71
devmor wrote: post #70

Earlier quoted context omitted.

>After training, we could take advantage of the structure to remove ~80% of the weights in topographic layers without sacrificing performance This is really interesting to me. Is it that the structure clustered the neurons in such a way that they didn't need to be weighted because their function were grouped by similar black box properties?

> Is it that the structure clustered the neurons in such a way that they didn't need to be weighted Yep. Because of the structure, we did not have to compute the output of each weight column and simply copied the outputs of nearby weight columns whose outputs were computed.

That is really cool and deserves the descriptor "brain-like", thank you for answering my questions!

Re: TopoNets: High performing vision and language models with brain-like topography

#75

Post by light_hue_1 »

mayukhdeb wrote: post #15
light_hue_1 wrote: post #9

They bury the part where inducing brain like structure hurts performance! This is a method to just hurt your network in exchange for nothing useful at all aside from some sketchy story that this is "brain like".

Our goal was never to optimize for performance. There's a long standing hypothesis that topographic structure in the human brain leads to metabolic efficiency. Thanks to topography in ANNs, we were able to test out this hypothesis in a computational setting. > sketchy story this is "brain like". we reproduce the hallmarks of functional organization seen in the visual and language cortex of the brain. I encourage you…

I did read the paper. I really hope I don't get assigned to be a reviewer for it.

You don't reproduce anything about the functional organization of the visual or language cortex. You make a pretty picture with blobs in it. And one that's trivial to get from current methods. If you think "the functional organization of the visual or language system" means random blobs of activation/connectivity, well, then it's time for a class on neuroscience. I cannot imagine what neuroscientist would let this fly reviewing the paper.

The whole "we don't optimize for performance" is nonsense. Take any modern method that prunes weights and beats your approach with ease. Then smooth its output a bit to make nice blobs. The performance loss from smoothing will still beat your method and look "brain-like" by your definition. There you go. Your experiments don't show anything at all aside from the fact that a bad method performs poorly.

You didn't think through controls or alternative hypotheses. You didn't take into account a decade of research on methods to prune networks. You don't take seriously what we know about functional organization in the brain.

All sorts of bad papers make it through reviewing these days. But.. you can definitely do better. Good luck!

Re: TopoNets: High performing vision and language models with brain-like topography

#76

Post by mayukhdeb »

devmor wrote: post #74
mayukhdeb wrote: post #71

Earlier quoted context omitted.

> Is it that the structure clustered the neurons in such a way that they didn't need to be weighted Yep. Because of the structure, we did not have to compute the output of each weight column and simply copied the outputs of nearby weight columns whose outputs were computed.

That is really cool and deserves the descriptor "brain-like", thank you for answering my questions!

Thanks for the kind words! Happy to know that there are people out there who find this stuff just as interesting as I do.
Post reply on HN