Neurons that fire together, wire together, but how? • Hacker News Board

Re: Neurons that fire together, wire together, but how?

#11

Post by dr_dshiv » Mon, Jun 29, 2020, 1:11 PM UTC

mongojunction wrote: post #4

What if something about the electrical signal attracts growth in certain direction, toward other signals firing at the same time? Or what if some neuron pairs that are not yet connected share quantum entangled structures, that if activated simultaneously ... but still how does direction occur? What if neurons emit light, that's why you can stimulate them with light...and what if they can somehow detect the faint ligh…

You are getting downvoted for speculating on Quantum entanglement, but I think all of your speculations are useful here and to be encouraged.

Re: Neurons that fire together, wire together, but how?

#12

Post by punnerud » Mon, Jun 29, 2020, 1:19 PM UTC

Main point: “(..) if the target neuron already has too many connections, it will tend to remove the weakest ones, and this includes the most recent ones. The scaling goes both ways after all – it goes for more synapses when it starts with too few, but for less, if it starts with too many.

But synaptic scaling is not everything. As it turns out, the tips of the growth cone constantly produce structures called filopodia, and these react to specific chemical attractants and repellents. These chemicals are produced by both cells at the target area, and by so-called guidepost cells along the way. There are suggestions that the system for such targeting is fairly robust, especially in early development (and its limitations in later life might explain why spinal cord injuries and the like are so hard to fix).“

Re: Neurons that fire together, wire together, but how?

#13

Post by dr_dshiv » Mon, Jun 29, 2020, 1:19 PM UTC

Hebb actually talked about causation, not synchrony (firing together):

"When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A ‘s efficiency, as one of the cells firing B, is increased”

Synchrony is extremely important, particularly for the formation of cortical columns and neural pruning. But in spike timing dependent plasticity, where growth is potentiated if the presynaptic fires just before post synaptic firing, the connection is actually depressed if up and downstream neurons fire exactly synchronously. (there is a huge amount of variation in this across the brain, though)

Re: Neurons that fire together, wire together, but how?

#14

Post by g_airborne » Mon, Jun 29, 2020, 1:23 PM UTC

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the network grows connections itself. The only counterpoint I can come up with is that is probably wouldn't generate a significant performance speed up because it defeats the use of SIMD/matrix operations on GPUs. Maybe we would need chips that are designed differently to speed up these self-growing networks?

I'm not an expert on this subject, does anybody have any insights on this?

1. https://www.technologyreview.com/2019/05/10/135426/a-new-way...

Re: Neurons that fire together, wire together, but how?

#15

Post by dr_dshiv » Mon, Jun 29, 2020, 1:32 PM UTC

dr_dshiv wrote: post #13

Hebb actually talked about causation, not synchrony (firing together): "When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A ‘s efficiency, as one of the cells firing B, is increased” Synchrony is extremely important, particularly for the formation of cortical columns and neur…

Note that there is also a mechanism for association between two presynaptic neurons. Probabilistically, when those upstream neurons fire synchronously, the downstream neuron will be more likely to actually fire. When that occurs, the postsynaptic neuron will, as a result of Hebb's Postulate, increase the connectivity to the synchronously firing neurons. So, "cells that fire together wire together" is more true of presynaptic neurons than pre-to-post synaptic neurons (and the wiring together is occurring through the postsynaptic neuron)

Re: Neurons that fire together, wire together, but how?

#16

Post by dr_dshiv » Mon, Jun 29, 2020, 1:49 PM UTC

When Hebb talks about "reverberation" in neural circuits, he still thinks in advance of our current knowledge of oscillatory neurodynamics. Here he speculates about the short term memory trace that holds position dynamically, prior to physical changes in the synapse:

"It might be supposed that the mnemonic trace is a lasting pattern of reverberatory activity without fixed locus like a cloud formation or eddies in a millpond"

From Hebb's 1948 "Organization of Behavior"

Re: Neurons that fire together, wire together, but how?

#17

Post by tjwhitaker » Mon, Jun 29, 2020, 2:05 PM UTC

g_airborne wrote: post #14

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…

I think this is a really interesting area of machine learning. Some efforts have been made in ideas that are tangential to this one. Lots of papers in neuroevolution deal with evolving topologies. NEAT is probably the prime example http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf and another paper I read recently called pathnet that is different but very interesting https://arxiv.org/abs/1701.08734.

Re: Neurons that fire together, wire together, but how?

#18

Post by Waterluvian » Mon, Jun 29, 2020, 2:07 PM UTC

jcims wrote: post #8

Mirioron wrote: post #2

This makes me want to know how quickly this type of growth happens. Is it on the order of seconds? Minutes? Hours? Days? Is this why when you learn something, take a break, come back later and everything makes more sense happens?

I've noticed that there's a weird area when learning a physical skill that there's a strange growth curve. You suck at first, then quickly get to some kind of milestone, then get worse before you get better. It feels like my brain is attempting to delegate some of the motor activity to lower levels before they are 'ready', but in fact it might be an essential part of training those neurons.

I think I get what you're saying. You _do_ get worse, then get better. It's like the brain is taking the proof of concept code and re-writing it with the necessary abstractions to make it a more performant routine. And then suddenly I start to experience the skills from new_task feeding into tasks I already know, and it feels like it's happening subconsciously.

Re: Neurons that fire together, wire together, but how?

#19

Post by [deleted] » Mon, Jun 29, 2020, 2:10 PM UTC

g_airborne wrote: post #14

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…

[deleted]

Re: Neurons that fire together, wire together, but how?

#20

Post by g_airborne » Mon, Jun 29, 2020, 2:11 PM UTC

tjwhitaker wrote: post #17

g_airborne wrote: post #14

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…

I think this is a really interesting area of machine learning. Some efforts have been made in ideas that are tangential to this one. Lots of papers in neuroevolution deal with evolving topologies. NEAT is probably the prime example http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf and another paper I read recently called pathnet that is different but very interesting https://arxiv.org/abs/1701.08734 .

This is very cool! Thanks!