TopoNets: High performing vision and language models with brain-like topography
1–10 of 76 posts
Re: TopoNets: High performing vision and language models with brain-like topography
#2Shouldn't there be a comparison in performance on common benchmarks to other models?
Like a 7B toponet model vs a 7B Llama model?
As a layperson I don't understand why topology is a thing to optimize for.
Re: TopoNets: High performing vision and language models with brain-like topography
#3[dead]
Re: TopoNets: High performing vision and language models with brain-like topography
#4The title here doesn't seem to match. The paper is called "TopoNets: High Performing Vision and Language Models with Brain-Like Topography"
Even with their new method, models with topography seem to perform worse than models without.
Re: TopoNets: High performing vision and language models with brain-like topography
#5The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. As compute and data grows, you want less inductive bias. For example, CNN will beat ViT on small data tasks, but that flips with enough scale because ViT imposes less inductive bias. Or at least any inductive bias should be chosen because it models the structure of the data well, such as with causal transformers and language.
Re: TopoNets: High performing vision and language models with brain-like topography
#6The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. A…
> So what's the motivation here?
Better interpretability, I suppose. Could give insights into how cognition works.
Re: TopoNets: High performing vision and language models with brain-like topography
#7Shouldn't there be a comparison in performance on common benchmarks to other models? Like a 7B toponet model vs a 7B Llama model? As a layperson I don't understand why topology is a thing to optimize for.
The only potential benefit shown in the paper is the topologically local models seem to be more resilient after pruning.
So you may be able to prune a 7B model down to 6B while maintaining most of the capability.
Re: TopoNets: High performing vision and language models with brain-like topography
#8The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. A…
Unless GPUs work markedly differently somehow or there’s been some fundamental shift in computer architecture I’m not aware of, spatial locality is still a factor in computers.
Aside from HW acceleration today, designs like Cebras would benefit heavily by reducing the amount of random access from accessing the weights (and thus freeing up cross-chip memory bandwidth for other things).
Re: TopoNets: High performing vision and language models with brain-like topography
#9They bury the part where inducing brain like structure hurts performance!
This is a method to just hurt your network in exchange for nothing useful at all aside from some sketchy story that this is "brain like".
Re: TopoNets: High performing vision and language models with brain-like topography
#10The main reason topography emerges in physical brains is because spatially distant connections are physically difficult and expensive in biological systems. Artificial neural nets have no such trade-off. So what's the motivation here? I can understand this might be a very good regularizer, so it could help with generalization error on small-data tasks. But hard to see why this should be on the critical path to AGI. A…
Unless GPUs work markedly differently somehow or there’s been some fundamental shift in computer architecture I’m not aware of, spatial locality is still a factor in computers. Aside from HW acceleration today, designs like Cebras would benefit heavily by reducing the amount of random access from accessing the weights (and thus freeing up cross-chip memory bandwidth for other things).
This makes me remember game developers back when games could still be played directly from the physical disc. They would often duplicate data to different parts of the disc, knowing that certain data would often be streamed from disc together, so that seek times were minimized.
But those game devs knew where everything was spatially on the disc, and how the data would generally be used during gameplay. It was consistent.
Do engineers have a lot of insight into how models get loaded spatially onto a given GPU at run time? Is this constant? Is it variable on a per GPU basis? I would think it would have to be.
Hard to optimize for this.