Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neurons that fire together, wire together, but how?

dissociativediaries.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Neurons that fire together, wire together, but how?

#31

Post by vinay427 »

ramraj07 wrote: post #25
vinay427 wrote: post #5

I find it strange that the author couldn't find this in a textbook. This is rather common material in a developmental neuroscience textbook or lecture. I've looked through two such books during my (only) course on the topic, and all three sources covered this material.

What did they say ? Perhaps you misread what was in the textbooks? My understanding is that the authors questions are legitimate and still not fully answered.

Yep, I agree that there are legitimate questions raised which still lack answers. However, I'm responding to the claim that the information the author provides in the summary is not found in any textbook they saw:

> So there you have it, a quick summary of one part of neural connectivity I’ve yet to see described in a textbook about the brain, but which really should be given out there, along with the classic Hebbian principle

Re: Neurons that fire together, wire together, but how?

#32

Post by g_airborne »

blamestross wrote: post #22
g_airborne wrote: post #14

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…

(See sibling comment NEAT is awesome) The only reason we architect ANNs the way we do is optimization of computation. The bipartite graph structure is optimized for GPU matrix math. Systems like NEAT have not been used at scale because they are a lot more expensive to train and to utilize the trained network with. ASICs and FPGAs have a change to utilize a NEAT generated network in production, but we still don't have…

So this might be an enormous opportunity for low-cost and more performant AI if someone was able to build an FPGA of some sort that could handle these types of computations as efficiently right?

Re: Neurons that fire together, wire together, but how?

#33

Post by blamestross »

g_airborne wrote: post #32
blamestross wrote: post #22

Earlier quoted context omitted.

(See sibling comment NEAT is awesome) The only reason we architect ANNs the way we do is optimization of computation. The bipartite graph structure is optimized for GPU matrix math. Systems like NEAT have not been used at scale because they are a lot more expensive to train and to utilize the trained network with. ASICs and FPGAs have a change to utilize a NEAT generated network in production, but we still don't have…

So this might be an enormous opportunity for low-cost and more performant AI if someone was able to build an FPGA of some sort that could handle these types of computations as efficiently right?

Running the post-training network is a solved problem. (FPGA and ASIC can do it just fine). TRAINING the network is the difficulty. The problem is that the structure of the network is arbitrary and is a result of the learning process. You can't optimize a computation for a structure you don't know yet. Bipartite layer networks have the benefit of never changing structure but they can approximate other subset structures. I don't know if we could easily tell where we are on the tradeoff between "bipartite graphs are trained efficiently but are inefficiently simulating a smaller network in practice"

Re: Neurons that fire together, wire together, but how?

#34

Post by Der_Einzige »

blamestross wrote: post #22
g_airborne wrote: post #14

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…

(See sibling comment NEAT is awesome) The only reason we architect ANNs the way we do is optimization of computation. The bipartite graph structure is optimized for GPU matrix math. Systems like NEAT have not been used at scale because they are a lot more expensive to train and to utilize the trained network with. ASICs and FPGAs have a change to utilize a NEAT generated network in production, but we still don't have…

NEAT just doesn't have good, modern GPU powered implementations.

NEAT would totally be competitive if someone actually gets a version running in PyTorch/Tensorflow

Re: Neurons that fire together, wire together, but how?

#35

Post by dr_dshiv »

buboard wrote: post #30
dr_dshiv wrote: post #10

Earlier quoted context omitted.

What about hippocampal neurogenesis? Those are spewing out at a nearly constant rate all the time

it's still a tiny amount of neurons that are being turned over. About 1.75% of the dentate gyrus is renewed per year.

But that's the piano roll that is recording our sense of time!

Re: Neurons that fire together, wire together, but how?

#36

Post by mennis16 »

g_airborne wrote: post #14

The connectedness of neurons in neural nets is usually fixed from the start (i.e. between layers, or somewhat more complicated in the case CNNs etc). If we could eliminate this and let neurons "grow" towards each other (like this article shows), would that enable smaller networks with similar accuracy? There's some ongoing research to prune weights by finding "subnets" [1] but I haven't found any method yet where the…

Here is a relevant paper, which was the coolest thing I saw at this past NeurIPS: https://weightagnostic.github.io/

It is based on NEAT (as other commenters mentioned) and also ties in some discussion of the Lottery Ticket Hypothesis as you mentioned.

Re: Neurons that fire together, wire together, but how?

#37

Post by jawarner »

Der_Einzige wrote: post #34
blamestross wrote: post #22

Earlier quoted context omitted.

(See sibling comment NEAT is awesome) The only reason we architect ANNs the way we do is optimization of computation. The bipartite graph structure is optimized for GPU matrix math. Systems like NEAT have not been used at scale because they are a lot more expensive to train and to utilize the trained network with. ASICs and FPGAs have a change to utilize a NEAT generated network in production, but we still don't have…

NEAT just doesn't have good, modern GPU powered implementations. NEAT would totally be competitive if someone actually gets a version running in PyTorch/Tensorflow

You may be interested in this implementation [1] which builds the networks using PyTorch.

[1] https://github.com/uber-research/PyTorch-NEAT

Re: Neurons that fire together, wire together, but how?

#38

Post by blamestross »

Der_Einzige wrote: post #34
blamestross wrote: post #22

Earlier quoted context omitted.

(See sibling comment NEAT is awesome) The only reason we architect ANNs the way we do is optimization of computation. The bipartite graph structure is optimized for GPU matrix math. Systems like NEAT have not been used at scale because they are a lot more expensive to train and to utilize the trained network with. ASICs and FPGAs have a change to utilize a NEAT generated network in production, but we still don't have…

NEAT just doesn't have good, modern GPU powered implementations. NEAT would totally be competitive if someone actually gets a version running in PyTorch/Tensorflow

It's not that simple. Backpropagating a bipartite graph of nodes works out to a series of matrix operations that parallelize efficiently on a GPU as long as the matrices fit into the GPU's working memory. Running a GA (part of neat) doesn't normally work well on a GPU. The good NEAT algorithms even allow different neurons to have different firing response curves. This inherently defies the "same operation multiple values" style of parallelization in GPUs. The way GPUs work just fundamentally isn't well suited to speeding up NEAT.

Re: Neurons that fire together, wire together, but how?

#39

Post by jcims »

mikhailfranco wrote: post #24
jcims wrote: post #8

Earlier quoted context omitted.

I've noticed that there's a weird area when learning a physical skill that there's a strange growth curve. You suck at first, then quickly get to some kind of milestone, then get worse before you get better. It feels like my brain is attempting to delegate some of the motor activity to lower levels before they are 'ready', but in fact it might be an essential part of training those neurons.

See Mastery by George Leonard which is a great book and highly recommended even if you are not into karate or martial arts. You have echoed his sketch of punctuated plateaus (p14): The Mastery Curve There's really no way around it. Learning any new skill involves relatively brief spurts of progress, each of which is followed by a slight decline to a plateau somewhat higher in most cases than that which preceded it. […

Will definitely check this out thank you

Re: Neurons that fire together, wire together, but how?

#40

Post by blamestross »

jawarner wrote: post #37
Der_Einzige wrote: post #34

Earlier quoted context omitted.

NEAT just doesn't have good, modern GPU powered implementations. NEAT would totally be competitive if someone actually gets a version running in PyTorch/Tensorflow

You may be interested in this implementation [1] which builds the networks using PyTorch. [1] https://github.com/uber-research/PyTorch-NEAT

It uses pytorch (and I'm probably going to use it), but doesn't effectively leverage a GPU for training.
Post reply on HN