So basically L2 regularization.
Not really. This is a local inhibition, and it's not clear if there's penalty on synapse growth.
Study reveals how, when a synapse strengthens, its neighbors weaken
21–30 of 46 posts
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#22Earlier quoted context omitted.
I'm still wondering about the natural equivalent of back-propagation.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3673183/ Citations here: https://scholar.google.com/scholar?cites=6395717759743355511
I think the term "backpropagation" has a different meaning in neurobiology.
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#23In deep learning this is called local response normalization (LRN). ConvNets use LRN where most active neurons inhibit other neurons at the same location in neighboring feature maps. http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
Actually the new finding is similar to weight normalisation for individual neurons. LRN would be similar to another thing called lateral inhibition.
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#24My naive take on this is that it makes biologically sense to keep net expected electrical impulses approximately the same before and after strengthening. At the end of the day the brain has energy constraints.
This can-be/is done functionally in ANNs but achieves a different end (avoids over-fitting) but doesn't reducing energy(compute) expenditure in dense ANNs since activation and non-activation is computed in expectation and take the same number of cycles in dense networks.
I'd love to see more work on massive sparse networks, where you actually get compute efficiency if you can reduce number of activation without reducing hurting your optimization target.
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#25The naive conclusion is to avoid learning in any form in order to keep all the synapses in optimal state. Rather disappointing.
[deleted]
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#26In deep learning this is called local response normalization (LRN). ConvNets use LRN where most active neurons inhibit other neurons at the same location in neighboring feature maps. http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
Actually the new finding is similar to weight normalisation for individual neurons. LRN would be similar to another thing called lateral inhibition.
[deleted]
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#27The naive conclusion is to avoid learning in any form in order to keep all the synapses in optimal state. Rather disappointing.
That's too naive. There's next to nothing known about how activity in individual neurons and their synapses relates to mental contents, in particular "higher level" concepts and thought patterns that you're concerned with in adult learning. In other words, the network dynamics are complex and unknown, and so the suppression / deactivation of certain synapses may just as well be a normal and necessary part of learning…
Amen to that, we are starting to get some of the groundwork but are far far away from a “standard model”, neuroscientists,psychiatrists and AI developers oversell their understanding in order to keep their jobs and funding.
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#28Earlier quoted context omitted.
That's too naive. There's next to nothing known about how activity in individual neurons and their synapses relates to mental contents, in particular "higher level" concepts and thought patterns that you're concerned with in adult learning. In other words, the network dynamics are complex and unknown, and so the suppression / deactivation of certain synapses may just as well be a normal and necessary part of learning…
Amen to that, we are starting to get some of the groundwork but are far far away from a “standard model”, neuroscientists,psychiatrists and AI developers oversell their understanding in order to keep their jobs and funding.
I know of no psychologist or neuroscientist who even try for a 'standard model' level of understanding of brain function.
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#29I wonder if the progress in artificial neural network technology is going to be correlated to discoveries on real neural networks.
I'm still wondering about the natural equivalent of back-propagation.
Isn't that mostly a supervised learning technique? Maybe the larger question is whether learning in the brain is better described as unsupervised or supervised.
Re: Study reveals how, when a synapse strengthens, its neighbors weaken
#30Does this mean addiction can completely be cured