Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neurons that fire together, wire together, but how?

dissociativediaries.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Neurons that fire together, wire together, but how?

#41

Post by joveian »

jcims wrote: post #8
Mirioron wrote: post #2

This makes me want to know how quickly this type of growth happens. Is it on the order of seconds? Minutes? Hours? Days? Is this why when you learn something, take a break, come back later and everything makes more sense happens?

I've noticed that there's a weird area when learning a physical skill that there's a strange growth curve. You suck at first, then quickly get to some kind of milestone, then get worse before you get better. It feels like my brain is attempting to delegate some of the motor activity to lower levels before they are 'ready', but in fact it might be an essential part of training those neurons.

A lot of learning involves interaction between the cerebellum and the rest of the brain, and the cerebellum has a different structure. "How the brain works" type descriptions should mention the cerebellum quite a lot, but usually don't. Partly because when I was looking at this stuff a while back there was much less known about this interaction.

Re: Neurons that fire together, wire together, but how?

#42

Post by jawarner »

blamestross wrote: post #40
jawarner wrote: post #37

Earlier quoted context omitted.

You may be interested in this implementation [1] which builds the networks using PyTorch. [1] https://github.com/uber-research/PyTorch-NEAT

It uses pytorch (and I'm probably going to use it), but doesn't effectively leverage a GPU for training.

What do you think is the best way to accomplish this?

Re: Neurons that fire together, wire together, but how?

#44

Post by pshc »

londons_explore wrote: post #26
g_airborne wrote: post #14

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…

I experimented with networks where weights were removed if they did not contribute much to the final answer. My conclusion was I could easily set >99% of weights to zero on my (fully connected) layers with minimal performance impact after enough training, but the training time went up a lot (effectively after removing a bunch of connections, you have to do more training before removing more), and inference speed wasn…

Would you say you found a result similar to the lottery ticket hypothesis? https://arxiv.org/abs/1803.03635

Re: Neurons that fire together, wire together, but how?

#45

Post by londons_explore »

pshc wrote: post #44
londons_explore wrote: post #26

Earlier quoted context omitted.

I experimented with networks where weights were removed if they did not contribute much to the final answer. My conclusion was I could easily set >99% of weights to zero on my (fully connected) layers with minimal performance impact after enough training, but the training time went up a lot (effectively after removing a bunch of connections, you have to do more training before removing more), and inference speed wasn…

Would you say you found a result similar to the lottery ticket hypothesis? https://arxiv.org/abs/1803.03635

Not really - I had to do multiple steps of 'prune a bit, train a bit' to be able to prune to 99%. If I had done all the pruning in one big step as they do, I don't think it would have trained well, even if I had been able to see the future and remove the same weights.

Re: Neurons that fire together, wire together, but how?

#46

Post by mongojunction »

dr_dshiv wrote: post #11
mongojunction wrote: post #4

What if something about the electrical signal attracts growth in certain direction, toward other signals firing at the same time? Or what if some neuron pairs that are not yet connected share quantum entangled structures, that if activated simultaneously ... but still how does direction occur? What if neurons emit light, that's why you can stimulate them with light...and what if they can somehow detect the faint ligh…

You are getting downvoted for speculating on Quantum entanglement, but I think all of your speculations are useful here and to be encouraged.

Thank you. so you work in this area? what do you think about the software available for your research? Could it be better or does software not play so much of a role?

Re: Neurons that fire together, wire together, but how?

#47

Post by dr_dshiv »

mongojunction wrote: post #46
dr_dshiv wrote: post #11

Earlier quoted context omitted.

You are getting downvoted for speculating on Quantum entanglement, but I think all of your speculations are useful here and to be encouraged.

Thank you. so you work in this area? what do you think about the software available for your research? Could it be better or does software not play so much of a role?

Yes the software in this area is critical -- and there are major challenges.

Re: Neurons that fire together, wire together, but how?

#48

Post by mongojunction »

dr_dshiv wrote: post #47
mongojunction wrote: post #46

Earlier quoted context omitted.

Thank you. so you work in this area? what do you think about the software available for your research? Could it be better or does software not play so much of a role?

Yes the software in this area is critical -- and there are major challenges.

If it's not too much trouble could you point me to some resources, compilations of relevant software? Maybe I could add some value...

Re: Neurons that fire together, wire together, but how?

#49

Post by blamestross »

jawarner wrote: post #42
blamestross wrote: post #40

Earlier quoted context omitted.

It uses pytorch (and I'm probably going to use it), but doesn't effectively leverage a GPU for training.

What do you think is the best way to accomplish this?

You don't. You need a different parallelism model than a GPU provides. It could work well on machines with very high CPU count, but the speedup on GPUs is the main reason bipartite graph algorithms have seen such investment.
Post reply on HN