Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neurogenesis Deep Learning

arxiv.org

Post reply on HN
Print view View original

51–60 of 105 posts

Re: Neurogenesis Deep Learning

#51

Post by hyperbovine »

throwaway287391 wrote: post #34
joantune wrote: post #10

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature

I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional…

> We have little to no idea how actual neurons (let alone entire brains) really work.

I think that slights neuroscience, which has devoted the past 60 years to answering this question, to a fair degree. But I agree that the biomimetic motivations offered up for various flavors of neural net feel pretty bogus. It seems to me like, among the major old-school researchers in the field, only Geoff Hinton still does this.

Re: Neurogenesis Deep Learning

#52

Post by joantune »

bmh100 wrote: post #46
joantune wrote: post #38

Earlier quoted context omitted.

I meant that it's a bit similar in the way that the passing on of signals and how over time neurons prefer some connections rather than others. This part is a bit similar, no?

In a very hand-wavy sense, yes. The same can be said of paths to food by ant colonies. The way that ANNs have been drawn as circles with arrows between them looks like a cartoon version of neurons and synapses, which is the origin of the "neural network" part. The timing of data from hidden node to hidden node, the activation functions, and the hidden node outputs have very little to do with biological neurons. ANNs…

Wasn't the attempt of modeling a collective of neurons, their synapsis, and the way that some connections are reinforced the genesis of the artificial neural networks? That's how the first person brought that concept to life, no? He didn't even have a theoretical explanation on how/why that would work for something, right?

> ANNs have more in common with a CPU than a brain

How so? which parts are similar?

Re: Neurogenesis Deep Learning

#54

Post by SubiculumCode »

They simulate neurogenesis, I guess, but they do not incorporate the most interesting part of that neurogenesis: That is the new neurons are born into the dentate gyrus, a region thought to have a particular capacity to orthoganalize feature representations that are similar (e.g. pattern separate) allowing distinct memories to be formed for similar events. The dentate gyrus outputs to a region called Cornu ammonis 3 (CA3) which is heavily recurrent, and thought to br able to pattern complete a full representation from partial inputs. That is, CA3 can encode and retrieve the relations between 2 or more features or objects. For a mathematical model and review one might read: Rolls (2013) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3812781/

but many others exist. I'd write more but typing in my phone is driving me to distraction.

Re: Neurogenesis Deep Learning

#55

Post by [deleted] »

amelius wrote: post #17
hmate9 wrote: post #8

Slightly off topic, but I hate how publications are written. It seems like authors are purposely using big words and sentences that are often 5-6 lines long in order to make it seem more clever. I find myself often having to reread a sentence in order to understand it. These algorithms are often very simple and can be easily explained. Don't over complicate them.

Then here's a challenge: could you write the abstract of the article in "simple English", without changing the meaning?

[deleted]

Re: Neurogenesis Deep Learning

#56

Post by [deleted] »

throwaway287391 wrote: post #34
joantune wrote: post #10

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature

I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional…

[deleted]

Re: Neurogenesis Deep Learning

#57

Post by sxg »

throwaway287391 wrote: post #34
joantune wrote: post #10

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature

I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional…

Actually we really know a lot about how neurons work. We've got the biophysical properties down, and we understand neurotransmission at the cellular/molecular level for a lot of different types of neurons. We understand signal processing where we transduce sound, smell, sight, touch, taste into neurochemical signals. We even know a decent amount about the early phases of the processing of these "raw data" signals into higher levels of abstraction (e.g. edge detection for vision). What we don't understand is the later phases of processing (advanced layers of abstraction) all the way up to conscious sensation.

Re: Neurogenesis Deep Learning

#58

Post by the8472 »

hmate9 wrote: post #44
shmageggy wrote: post #25

Earlier quoted context omitted.

Although I just glanced at a few parts of this, I did not find it to be poorly written. Can you give an example where you thought it was too verbose or unnecessarily complex?

For example in the abstract: "adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations" Nobody talks like this. In my head I read this sentence and I have to translate it to "we add extra neurons to existing networks so they can learn new information while remembering everything it already kno…

> Nobody talks like this.

> I have to translate it

But you have to distinguish active and passive vocabulary. Words you commonly use and words you understand when others use them. And you also have to distinguish between written and spoken language. E.g. nobody would actually say "exempli gratia" but it's commonly used in writing.

English is not my mother tongue and for me it mostly falls into passive vocabulary, but it is perfectly understandable. I don't have to mentally translate the whole sentence into simpler words before being able to grasp its meaning.

And it's not like those words are obscure, they are just place 2 or 3 of most common uses among their respective set of synonyms.

Re: Neurogenesis Deep Learning

#59

Post by throwaway287391 »

hyperbovine wrote: post #51
throwaway287391 wrote: post #34

Earlier quoted context omitted.

I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional…

> We have little to no idea how actual neurons (let alone entire brains) really work. I think that slights neuroscience, which has devoted the past 60 years to answering this question, to a fair degree. But I agree that the biomimetic motivations offered up for various flavors of neural net feel pretty bogus. It seems to me like, among the major old-school researchers in the field, only Geoff Hinton still does this.

Fair. I was definitely unnecessarily harsh on neuroscience; my quibble is only with my own community's claims that what we're doing is anything like how the brain works. Thanks to you and sxg for correcting the record.

Re: Neurogenesis Deep Learning

#60

Post by habitue »

argonaut wrote: post #50

Sorry if I'm being snobbish, but I do wonder why this paper is only being submitted to IJCNN, a 2nd tier machine learning conference. I know students who publish undergrad research at workshops with lower acceptance rates than IJCNN. I can't think of any important machine learning papers published in IJCNN in the recent past.

It depends on what conclusions you're trying to draw from that information. What conference a paper was accepted to is a second-order signal of the noteworthiness. It's probably easier for someone versed in the field to just read the paper to determine if it's interesting. If you're using the conference as a quick pass/fail as you skim through the abstracts of hundreds of papers, ok, but you probably wouldn't make time to comment on HN about it in that case.

This paper looks like it builds on pretty well-known techniques like stacked autoencoders, so let's see what first-order noteworthiness data we can gather from a quick skim of the paper. If I had to guess why it wasn't accepted into a better conference:

- It uses stacked autoencoders, which are pretty out of fashion

- It bothers reporting results on MNIST

- (more subjectively) It pulls an unfortunately common technique of saying "here's something the brain does" and then hand-waving that it's a deep reason why a technique they've come up with is useful, when in fact the relationship is just "inspired by the general idea of", not "performs the same function as" the biological mechanism. In this case, I think the tenuous connection of their technique to research on neurogenesis is pretty flimsy. Clearly neurogenesis is not how an adult human brain forms new memories or gains proficiency in new skills (which they acknowledge in the conclusion)

Post reply on HN