How does this relate to Progressive Neural Networks [0]? That technique is also about accumulating knowledge (while not forgetting existing knowledge)
Neurogenesis Deep Learning
31–40 of 105 posts
Re: Neurogenesis Deep Learning
#32It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature
Well, neurons have many properties... their information processing capabilities are one aspect, but they also deal with the physical level of communication and staying healthy.
Neurons are also a family of cells, and are very diverse in shapes and functions. We tend to oversimplify our representation of neurons. There are simple neurons and then you have neurons like the Purkinje cell that are massive.
Neurons also rely on their counterparts, the glial cells, that are much less often mentioned.
I think because of this, it will be a while until we fully understand the role of each one of them.
Re: Neurogenesis Deep Learning
#33Slightly off topic, but I hate how publications are written. It seems like authors are purposely using big words and sentences that are often 5-6 lines long in order to make it seem more clever. I find myself often having to reread a sentence in order to understand it. These algorithms are often very simple and can be easily explained. Don't over complicate them.
Then here's a challenge: could you write the abstract of the article in "simple English", without changing the meaning?
Here's my attempt; not a huge number of changes because it was not too bad to begin with, but with slightly less self-indulgent language, and a lot of the jargon has to stay (partly because I don't know the field):
Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing - areas where humans have always performed better than conventional algorithms. In contrast to biological neural systems, which are capable of learning continuously, deep artificial networks have a limited ability for incorporating new information after a network has been trained. As a result, continuous learning methods could be very helpful in allowing deep networks to handle data sets which change over time. Here, inspired by the process of adult neurogenesis in the hippocampus, we investigate how adding new neurons to artificial neural networks can allow them to acquire new information, while preserving what they have already learned. Our results on the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes lower and upper case letters and digits, show that neurogenesis looks like a good approach for tackling the "stability-plasticity dilemma" that has been a problem for adaptive machine learning algorithms for some time.
As an academic, I tend to agree that we frequently feel compelled to apply more verbosity than is strictly required in order to communicate the intended semantic constructs.
Re: Neurogenesis Deep Learning
#34It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature
I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional networks".
Re: Neurogenesis Deep Learning
#35Neurogensis? How about neural death as a way to prune large neural networks into more compact ones--now that is a research idea!
Looks like these researchers are trying to make a network more adaptive, I think that deleting nodes would only make them worse at the current task they're being trained on as well as worse on the tasks they're being adapted to. You could train a model using neurogenesis to increase its accuracy, and then use distillation to train a smaller network to comparable accuracy. But these are two very different, but complem…
You're assuming that all nodes are functionally important/non-spurious.
Re: Neurogenesis Deep Learning
#36Very wishful thinking on my part, but I think we're far closer to a general intelligence than most expect.
I think what we might see is a kind of autonomous corporation that is nominally under the control of shareholders, a CEO or a board, but which makes decisions without very much or any human input, and which gains some amount of legal rights through corporate personhood.
It won't be a 'general ai', though. More like a set of loosely connected systems that operate 'in the best interests of the shareholders', however that's defined.
It's pretty much the end state of the trend of pushing decision making to algorithms to remove moral and legal culpability from individuals.
Re: Neurogenesis Deep Learning
#37Slightly off topic, but I hate how publications are written. It seems like authors are purposely using big words and sentences that are often 5-6 lines long in order to make it seem more clever. I find myself often having to reread a sentence in order to understand it. These algorithms are often very simple and can be easily explained. Don't over complicate them.
A lot of the time the verbosity isn't so much to sound more clever as it is to be very specific and explicit about what the author is trying to convey. There's a lot of changing assumed knowledge and jargon in various fields and our use of language changes over time. The publication writing style is an attempt to factor that out.
That's what an appendix is for. Letters to Nature are 1500 words. You'd be surprised how effective that forcing function is. Not only do you preserve meaning, but you can convey it better because it respects the readers cache size limitations.
Re: Neurogenesis Deep Learning
#38It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature
I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional…
I meant that it's a bit similar in the way that the passing on of signals and how over time neurons prefer some connections rather than others.
This part is a bit similar, no?
Re: Neurogenesis Deep Learning
#39It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature
This isn't strictly true though. Spiking Neural Networks [0] attempt to be more accurate representations of human neurons, but haven't really caught on because they aren't really much better than our perceptron model of neurons, at least for the things we are trying to do with them. [0] http://www.ane.pl/pdf/7146.pdf
nice, thanks for sharing, interesting read so far (read the introduction), will definitely give it a better look out of curiosity
Re: Neurogenesis Deep Learning
#40Earlier quoted context omitted.
I would have said 30 years, and I thought I was the optimist :) Anyway I hope you're right and I'm wrong!
Compared to researchers in the field, even your prediction is very optimistic.
Can you provide references ? I'd like to know what researchers think about it.