Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neurogenesis Deep Learning

arxiv.org

Post reply on HN
Print view View original

81–90 of 105 posts

Re: Neurogenesis Deep Learning

#81

Post by chriswarbo »

jostmey wrote: post #3

Neurogensis? How about neural death as a way to prune large neural networks into more compact ones--now that is a research idea!

As others have mentioned, there are approaches like regularisation and dropout which try to do similar things. What I find interesting is the fact there are two reasons to do this: to generalise/avoid-overfitting and to reduce resource usage.

It seems like almost all effort is spent on the former, since everyone's aiming for higher accuracy numbers. Are there any widely-used methods to tackle the latter?

For example, I'm imagining a system which is either given measurements of its resource usage (time, memory, etc.) or uses some simple predictive model (e.g. time ~ number of layers * some constant), and works within some resource bound:

- If we're below the bound, expand the model (add neurons, etc.) to allow accuracy increases (note "allow": it's ok to ignore/regularise-to-zero the extra parameters to avoid overfitting)

- If we're above the bound, prune the model (in a way which tries to preserve accuracy)

- Allocate resources to optimise some objective, e.g. reduce variance by pruning the parameters of the best-performing class/predictor/etc. and using those resources to expand the worst performer.

The closest thing I know of are artificial economies, but they seem to be more like a selection mechanism (akin to genetic programming) than a direct optimisation procedure (like gradient descent on an ANN).

Re: Neurogenesis Deep Learning

#82

Post by lawless123 »

bmh100 wrote: post #62
joantune wrote: post #52

Earlier quoted context omitted.

Wasn't the attempt of modeling a collective of neurons, their synapsis, and the way that some connections are reinforced the genesis of the artificial neural networks? That's how the first person brought that concept to life, no? He didn't even have a theoretical explanation on how/why that would work for something, right? > ANNs have more in common with a CPU than a brain How so? which parts are similar?

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

> Instantaneous transmission of signals vs. delay imposed by axon and dendrite length

Would there be anything to gain by simulating this?

Re: Neurogenesis Deep Learning

#83

Post by joantune »

bmh100 wrote: post #62
joantune wrote: post #52

Earlier quoted context omitted.

Wasn't the attempt of modeling a collective of neurons, their synapsis, and the way that some connections are reinforced the genesis of the artificial neural networks? That's how the first person brought that concept to life, no? He didn't even have a theoretical explanation on how/why that would work for something, right? > ANNs have more in common with a CPU than a brain How so? which parts are similar?

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

Sure, they are still running on CPUs, but ANNs are still modeled with CPUs to do what NNs do, at least at some levels where experiments showed that they work.

Sure, some of the properties of NNs do not transpose well to ANNs. As someone pointed out in a comment here with an article showing that if you apply the same kind of signal it doesn't work.

But the fact remains: we are being more successful on AI advancements by trying to emulate parts of our brain than we were with other techniques.

We didn't knew that this would happen when it all started, but it did.

Out of the blue, no-one could look at a model of a yet to be implemented ANN and say it would work, and why. It has all been experimentation, taking the brain as a raw blueprint.

And although many other phenomenas that happened with the brain didn't work well with ANNs, neurogenesis apparently did.

It's impressive IMO and quite humbling that we are getting so many achievements out from mimicking nature, and we aren't 100% sure why it worked in the first place.

That's all that I meant to say

Re: Neurogenesis Deep Learning

#84

Post by AstralStorm »

lawless123 wrote: post #82
bmh100 wrote: post #62

Earlier quoted context omitted.

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

> Instantaneous transmission of signals vs. delay imposed by axon and dendrite length Would there be anything to gain by simulating this?

Temporal recurrent neural networks have been tried, I think by Microsoft Research.

Re: Neurogenesis Deep Learning

#85

Post by AstralStorm »

bmh100 wrote: post #62
joantune wrote: post #52

Earlier quoted context omitted.

Wasn't the attempt of modeling a collective of neurons, their synapsis, and the way that some connections are reinforced the genesis of the artificial neural networks? That's how the first person brought that concept to life, no? He didn't even have a theoretical explanation on how/why that would work for something, right? > ANNs have more in common with a CPU than a brain How so? which parts are similar?

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

Also:

* local regular structure vs irregular structure with global elements

Re: Neurogenesis Deep Learning

#86

Post by AstralStorm »

bmh100 wrote: post #62
joantune wrote: post #52

Earlier quoted context omitted.

Wasn't the attempt of modeling a collective of neurons, their synapsis, and the way that some connections are reinforced the genesis of the artificial neural networks? That's how the first person brought that concept to life, no? He didn't even have a theoretical explanation on how/why that would work for something, right? > ANNs have more in common with a CPU than a brain How so? which parts are similar?

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

Also:

* local regular structure vs irregular structure with global elements

Re: Neurogenesis Deep Learning

#87

Post by visarga »

chriswarbo wrote: post #81
jostmey wrote: post #3

Neurogensis? How about neural death as a way to prune large neural networks into more compact ones--now that is a research idea!

As others have mentioned, there are approaches like regularisation and dropout which try to do similar things. What I find interesting is the fact there are two reasons to do this: to generalise/avoid-overfitting and to reduce resource usage. It seems like almost all effort is spent on the former, since everyone's aiming for higher accuracy numbers. Are there any widely-used methods to tackle the latter? For example,…

There are many ways to compress networks - by pruning neurons, by enforcing sparsity, by representing activations and gradients on one bit (or a few bits), and by transfer learning where a large net is transferred into a smaller one.

Re: Neurogenesis Deep Learning

#88

Post by visarga »

antome wrote: post #5
jostmey wrote: post #3

Neurogensis? How about neural death as a way to prune large neural networks into more compact ones--now that is a research idea!

This idea is at least partially in use with regularisation and dropout. The difference at least with dropout is that the "killed" neurons are then massaged back into the network in order become useful again.

Biological neurons themselves are stochastic so they have an internal "dropout" that doesn't seem to hurt, on the contrary, these perturbations and imperfect communication increase learning ability.

Re: Neurogenesis Deep Learning

#89

Post by visarga »

joantune wrote: post #10

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature

We're not simulating brain neurons:

- real neurons are stochastic and communicate through spikes, artificial neurons can communicate real values efficiently

- real neurons are more like automatons, they have a dynamic in time, learning happens as a continuous interaction with only its neighbors; artificial neurons are "static" (use discrete time) and implemented by forward and backward pass, and also can use nonlocal information

- real neurons can't backpropagate, because backprop requires the transmission of gradients back the same connections, but in reverse - brain connections don't support that kind of bidirectional data flow; artificial neurons work best by backprop

- real neurons can't implement convolutions, it would require a neuron to slide over a field; also real neurons can't implement RNNs as they are, and don't use backpropagation through time BPTT

So, artificial neurons are much less hampered and can do many things that real neurons can't do or have to use some less efficient method. That means brain neurons still have some tricks up their sleeve. Artificial neurons are quite different from brain neurons, and it's right to be so, because they can be more efficient that way.

Re: Neurogenesis Deep Learning

#90

Post by chriswarbo »

visarga wrote: post #87
chriswarbo wrote: post #81

Earlier quoted context omitted.

As others have mentioned, there are approaches like regularisation and dropout which try to do similar things. What I find interesting is the fact there are two reasons to do this: to generalise/avoid-overfitting and to reduce resource usage. It seems like almost all effort is spent on the former, since everyone's aiming for higher accuracy numbers. Are there any widely-used methods to tackle the latter? For example,…

There are many ways to compress networks - by pruning neurons, by enforcing sparsity, by representing activations and gradients on one bit (or a few bits), and by transfer learning where a large net is transferred into a smaller one.

Yes, my question was more about meta-level algorithms for balancing size against performance. Especially adaptive methods such that we're not just growing up to a limit and stopping, but selectively allocating resources to those parts which need them. Adapting over time would be nice too: "thinking harder" when there are idle resources, but shrinking the results back down under load.
Post reply on HN