21–30 of 49 posts
The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…
The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…
The only reason we architect ANNs the way we do is optimization of computation. The bipartite graph structure is optimized for GPU matrix math. Systems like NEAT have not been used at scale because they are a lot more expensive to train and to utilize the trained network with. ASICs and FPGAs have a change to utilize a NEAT generated network in production, but we still don't have a computer well suited to training a NEAT network.
This makes me want to know how quickly this type of growth happens. Is it on the order of seconds? Minutes? Hours? Days? Is this why when you learn something, take a break, come back later and everything makes more sense happens?
I've noticed that there's a weird area when learning a physical skill that there's a strange growth curve. You suck at first, then quickly get to some kind of milestone, then get worse before you get better. It feels like my brain is attempting to delegate some of the motor activity to lower levels before they are 'ready', but in fact it might be an essential part of training those neurons.
which is a great book and highly recommended
even if you are not into karate or martial arts.
You have echoed his sketch of punctuated plateaus (p14):
The Mastery Curve
There's really no way around it. Learning any new skill
involves relatively brief spurts of progress, each of
which is followed by a slight decline to a plateau
somewhat higher in most cases than that which preceded it.
I find it strange that the author couldn't find this in a textbook. This is rather common material in a developmental neuroscience textbook or lecture. I've looked through two such books during my (only) course on the topic, and all three sources covered this material.
The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…
My conclusion was I could easily set >99% of weights to zero on my (fully connected) layers with minimal performance impact after enough training, but the training time went up a lot (effectively after removing a bunch of connections, you have to do more training before removing more), and inference speed wasn't really improved because sparse matrices are sloooow.
Overall, while it works out for biology, I don't think it will work for silicon.
This makes me want to know how quickly this type of growth happens. Is it on the order of seconds? Minutes? Hours? Days? Is this why when you learn something, take a break, come back later and everything makes more sense happens?
enhance transitive closure on a temporal window
plus the dual negation, whatever that is
under the space-time corollary of De Morgan's Laws:
atrophy atemporal uncorrelated direct connection
Growth cones are only relevant during development and in regenerating neurons which are not common. Everyday neurons do however continuously extend (and contract) filopodia which may reach nearby axon terminals and eventually form a synapse, thus causing synaptic rewiring. Synaptic scaling is usually used to refer to a homeostatic and uniforum up- or down- scaling of synaptic weights and is not really relevant to rew…
What about hippocampal neurogenesis? Those are spewing out at a nearly constant rate all the time