Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neurogenesis Deep Learning

arxiv.org

Post reply on HN
Print view View original

41–50 of 105 posts

Re: Neurogenesis Deep Learning

#41

Post by joantune »

spynxic wrote: post #16
joantune wrote: post #10

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature

I find that somewhat strange. Why attribute the idea of introducing new nodes to a graph to biological concepts? It seems like a simple step in exploration, similar to how one might think to vary the weights of the nodes randomly over some range.. unless there is some technique biology uses to pre-configure the nodes upon introduction to the network, that might be rather interesting.

Because they tried to model neurogenesis, the same way that artificial neural networks were invented while trying to mimic some parts of how neurons work I guess..

Re: Neurogenesis Deep Learning

#42

Post by hmate9 »

amelius wrote: post #17
hmate9 wrote: post #8

Slightly off topic, but I hate how publications are written. It seems like authors are purposely using big words and sentences that are often 5-6 lines long in order to make it seem more clever. I find myself often having to reread a sentence in order to understand it. These algorithms are often very simple and can be easily explained. Don't over complicate them.

Then here's a challenge: could you write the abstract of the article in "simple English", without changing the meaning?

Problem: current methods for training neural networks cannot learn continuously. Once they have been trained, they cannot easily learn new information. We propose a method to fix this problem. Our algorithm adds new neurons to the neural network which allows it to learn new information, while maintaining everything it already "knows".

I showed this to my father, a surgeon, and he said he understood it. But not the original abstract.

Re: Neurogenesis Deep Learning

#43

Post by joantune »

badminton1 wrote: post #32
joantune wrote: post #10

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature

Well, neurons have many properties... their information processing capabilities are one aspect, but they also deal with the physical level of communication and staying healthy. Neurons are also a family of cells, and are very diverse in shapes and functions. We tend to oversimplify our representation of neurons. There are simple neurons and then you have neurons like the Purkinje cell that are massive. Neurons also r…

If i'm not mistaken (I had the introductory class on Neural networks quite some years ago) this all started out of trying to simulate neurons in a manner.

Much so that one of the pioneers of this got discredited by other scientist that for some reason simply could not accept that these would work, and that same pioneer started to get his funding discredited and started believing in his opposition so much so that he sailed away to his death, some argue intentionally (as his life's work had been, even by his eyes, seen as useless).

Now, I'm having a hard time remembering the names of the people on that story, if someone knows who I'm talking about, please remind me of those

Re: Neurogenesis Deep Learning

#44

Post by hmate9 »

shmageggy wrote: post #25
hmate9 wrote: post #8

Slightly off topic, but I hate how publications are written. It seems like authors are purposely using big words and sentences that are often 5-6 lines long in order to make it seem more clever. I find myself often having to reread a sentence in order to understand it. These algorithms are often very simple and can be easily explained. Don't over complicate them.

Although I just glanced at a few parts of this, I did not find it to be poorly written. Can you give an example where you thought it was too verbose or unnecessarily complex?

For example in the abstract: "adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations"

Nobody talks like this. In my head I read this sentence and I have to translate it to "we add extra neurons to existing networks so they can learn new information while remembering everything it already knows".

Re: Neurogenesis Deep Learning

#45

Post by 10b5-1 »

rdlecler1 wrote: post #35
10b5-1 wrote: post #4

Earlier quoted context omitted.

Looks like these researchers are trying to make a network more adaptive, I think that deleting nodes would only make them worse at the current task they're being trained on as well as worse on the tasks they're being adapted to. You could train a model using neurogenesis to increase its accuracy, and then use distillation to train a smaller network to comparable accuracy. But these are two very different, but complem…

You're assuming that all nodes are functionally important/non-spurious.

I'm not assuming that, I'm giving the model more options and letting it decide what is functionally important/non-spurious. It might take it longer, but I don't assume that.

Re: Neurogenesis Deep Learning

#46

Post by bmh100 »

joantune wrote: post #38
throwaway287391 wrote: post #34

Earlier quoted context omitted.

I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional…

I meant that it's a bit similar in the way that the passing on of signals and how over time neurons prefer some connections rather than others. This part is a bit similar, no?

In a very hand-wavy sense, yes. The same can be said of paths to food by ant colonies. The way that ANNs have been drawn as circles with arrows between them looks like a cartoon version of neurons and synapses, which is the origin of the "neural network" part. The timing of data from hidden node to hidden node, the activation functions, and the hidden node outputs have very little to do with biological neurons. ANNs have more in common with a CPU than a brain.

Re: Neurogenesis Deep Learning

#47

Post by kajecounterhack »

jostmey wrote: post #3

Neurogensis? How about neural death as a way to prune large neural networks into more compact ones--now that is a research idea!

https://arxiv.org/abs/1503.02531

Modern applications of small networks regularly reduce sizes from larger state-of-the-art networks using distillation. Distillation compacts neural networks while affecting accuracy minimally.

Instead of pruning directly from the large network, just learn how it generalizes. Takes fewer nodes / overall operations (Multiplications / Additions).

Re: Neurogenesis Deep Learning

#49

Post by bmh100 »

hmate9 wrote: post #44
shmageggy wrote: post #25

Earlier quoted context omitted.

Although I just glanced at a few parts of this, I did not find it to be poorly written. Can you give an example where you thought it was too verbose or unnecessarily complex?

For example in the abstract: "adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations" Nobody talks like this. In my head I read this sentence and I have to translate it to "we add extra neurons to existing networks so they can learn new information while remembering everything it already kno…

However, your statement is actually vague and ambiguous.

"Extra neurons"? Input layer? Output layer? Just before a final, fully-connected layer? Somewhere in between?

"Everything it already knows"? What does it know? Character probabilities, like a charnn? Image categories like in a CNN? Input distributions, like a GAN?

From the abstract, I can immediately tell that this paper is about modifying deep auto-encoders in the hidden layers. From that, I can immediately understand that the paper is not about adapting to new input formats or output formats, but instead about inputs from a new distribution but in the same format.

The author's intended audience, researchers and academics, do talk like this. They do so because it is quickly understandable and actually information dense, as indicated in my above paragraph.

Re: Neurogenesis Deep Learning

#50

Post by argonaut »

Sorry if I'm being snobbish, but I do wonder why this paper is only being submitted to IJCNN, a 2nd tier machine learning conference. I know students who publish undergrad research at workshops with lower acceptance rates than IJCNN. I can't think of any important machine learning papers published in IJCNN in the recent past.
Post reply on HN