Earlier quoted context omitted.
Unfortunately dendritic compartmentalization, spike timing etc are still not present. All efforts at models of SNNs that I know of have hit problems like riddled basins so far, that is what to look for to move past the limits of perceptron based networks IMHO. As PAC learning with autograd and perceptrons is just compression, or set shattering, this paper is more of an optimization method that reduces ANN expressiven…
> reduces ANN expressiveness But does it? It’s been my hypothesis for a while that every grad-trained NN is hauling around a lot of “nascent” nodes — nodes that were on their way to being useful, but haven’t received enough input yet to actually have their outputs be distinguishable from noise / ever influence the output. Sort of the neuroplastic equivalent of an evolutionary pre-adaptation . If such nodes exist in N…
Consider when you use 'partial connectivity', E.G. convolution or pooling layers for local feature extraction on say MNIST.
While useful, those partial connection layers are explicitly used because fully connected layers do not have translational invariance.
So with a fully connected network, shifting the letter 'i' a few pixels to the right wouldn't match.
We choose to discard some of those connections for local feature detection. But as the reason that the fully connected model lacks translational invariance is because it maintains that position data.
Note how that is more 'expressive', even if counterproductive for the actual use case.
Another lens is the fact that neural networks have extreme simplicity bias. In that they learn only the simplest features to solve a task at hand.
If you want to recognize an i, irrespective of the translational location, that bias is useful. But you 'throw away' (in a very loose sense) the positional data to do so.
Horses for courses, not good vs bad.