Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neurogenesis Deep Learning

arxiv.org

Post reply on HN
Print view View original

91–100 of 105 posts

Re: Neurogenesis Deep Learning

#91

Post by SubiculumCode »

arkymark wrote: post #77
SubiculumCode wrote: post #54

They simulate neurogenesis, I guess, but they do not incorporate the most interesting part of that neurogenesis: That is the new neurons are born into the dentate gyrus, a region thought to have a particular capacity to orthoganalize feature representations that are similar (e.g. pattern separate) allowing distinct memories to be formed for similar events. The dentate gyrus outputs to a region called Cornu ammonis 3…

this is really interesting, where/how did you learn this? I'd like to learn more about these things - brain regions, connections, functions - and what they might imply about the kinds of computations that are going on, but my background is mainly on the AI/math side of things.

My dissertational research was on the development of the subfields of the hippocampus in childhood. So these papers from the rat literature were relevant and often inspiring.

Re: Neurogenesis Deep Learning

#92

Post by SubiculumCode »

arkymark wrote: post #77
SubiculumCode wrote: post #54

They simulate neurogenesis, I guess, but they do not incorporate the most interesting part of that neurogenesis: That is the new neurons are born into the dentate gyrus, a region thought to have a particular capacity to orthoganalize feature representations that are similar (e.g. pattern separate) allowing distinct memories to be formed for similar events. The dentate gyrus outputs to a region called Cornu ammonis 3…

this is really interesting, where/how did you learn this? I'd like to learn more about these things - brain regions, connections, functions - and what they might imply about the kinds of computations that are going on, but my background is mainly on the AI/math side of things.

I'd like to add that our knowledge of the details of hippocampal neuroanatomy are probably the most advanced of any brain region, which allows the somewhat informed construction of computational models. I wish I had more time to learn modeling methods, I have specific developmental hypotheses I'd like to test in such a model. In the end, I'd probably need to find a knowledgable collaborator though.

Re: Neurogenesis Deep Learning

#93

Post by Filligree »

gallerdude wrote: post #23
Filligree wrote: post #15

Earlier quoted context omitted.

That's not a thing to hope for. We haven't solved the value alignment problem, so the longer until we have human or better-than-human AI, the better.

Theory - does empathy rise with intelligence?

There's no reason to think it does, and plenty of reason to think it shouldn't. We have a fairly good reason of why humans have empathy; it has to do with our evolution as a social species.

Re: Neurogenesis Deep Learning

#94

Post by sharemywin »

gallerdude wrote: post #7

Very wishful thinking on my part, but I think we're far closer to a general intelligence than most expect.

general intelligence seems way over rated to me...

I have plenty of wants and desires that could take a whole army of idiot savants working 24/7 to fulfill.

Re: Neurogenesis Deep Learning

#95

Post by infinite8s »

arkymark wrote: post #77
SubiculumCode wrote: post #54

They simulate neurogenesis, I guess, but they do not incorporate the most interesting part of that neurogenesis: That is the new neurons are born into the dentate gyrus, a region thought to have a particular capacity to orthoganalize feature representations that are similar (e.g. pattern separate) allowing distinct memories to be formed for similar events. The dentate gyrus outputs to a region called Cornu ammonis 3…

this is really interesting, where/how did you learn this? I'd like to learn more about these things - brain regions, connections, functions - and what they might imply about the kinds of computations that are going on, but my background is mainly on the AI/math side of things.

One place to start is the "Principles of Neural Science", the intro textbook to neuroscience.

Re: Neurogenesis Deep Learning

#96

Post by kajecounterhack »

Vitrified wrote: post #79
kajecounterhack wrote: post #47

Earlier quoted context omitted.

https://arxiv.org/abs/1503.02531 Modern applications of small networks regularly reduce sizes from larger state-of-the-art networks using distillation. Distillation compacts neural networks while affecting accuracy minimally. Instead of pruning directly from the large network, just learn how it generalizes. Takes fewer nodes / overall operations (Multiplications / Additions).

Now that is interesting. I hadn't realized methods to combine trained networks so efficiently were already readily available.

Certain companies use these methods to make state of the art neural nets work on your phones :)

Also "combine" might not be the right word, since it's really transfer learning. "Distill" is really a descriptive verb.

Maybe my original wording was confusing; I shouldn't have said "distillation compacts" -- distillation is a process by which you can create a more compact version of a complex neural net.

Re: Neurogenesis Deep Learning

#97

Post by bmh100 »

joantune wrote: post #83
bmh100 wrote: post #62

Earlier quoted context omitted.

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

Sure, they are still running on CPUs, but ANNs are still modeled with CPUs to do what NNs do, at least at some levels where experiments showed that they work. Sure, some of the properties of NNs do not transpose well to ANNs. As someone pointed out in a comment here with an article showing that if you apply the same kind of signal it doesn't work. But the fact remains: we are being more successful on AI advancements…

I just don't want you to get the wrong impression. This is a single paper about a technique for adding neurons to ANNs over time, and it is only one of many over the last few decades. The paper does not have the evidence to indicate that this is a major breakthrough. The industry as a whole generally does not add neurons to an existing model when updating that model. The vast majority of applications also use backpropagation for training, which is not what our brains use. So even if we ignore the implementation on CPUs, ANNs are still far from behaving similarly, even in a conceptual way, from brains. I must disagree that "we are getting so many achievements out from mimicking nature".

Re: Neurogenesis Deep Learning

#98

Post by bmh100 »

AstralStorm wrote: post #85
bmh100 wrote: post #62

Earlier quoted context omitted.

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

Also: * local regular structure vs irregular structure with global elements

Oh yes. If we want to put a bigger one on the list, then there's the whole matter of vast quantities of circumstantial data being left out by ANNs (sight, sound, past memories, emotions, arousal, touch sensations, etc.). But lists like this can go on for very long.

Re: Neurogenesis Deep Learning

#99

Post by bmh100 »

lawless123 wrote: post #82
bmh100 wrote: post #62

Earlier quoted context omitted.

Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs…

> Instantaneous transmission of signals vs. delay imposed by axon and dendrite length Would there be anything to gain by simulating this?

It adds an additional parameter that influences RNN behavior over time, so I could see it possibly being useful. I would speculate that this could have value for providing slowly-updating subsystem information to real-time control systems.

Re: Neurogenesis Deep Learning

#100

Post by [deleted] »

chriswarbo wrote: post #80
thibauts wrote: post #40

Earlier quoted context omitted.

Can you provide references ? I'd like to know what researchers think about it.

There are a few sources at https://intelligence.org/all-publications/ Click on "FC Forecasting" near the top to limit the list to those about AI predictions, including: - The Errors, Insights and Lessons of Famous AI Predictions - and What They Mean for the Future - Predicting AGI: What Can We Say When We Know So Little? - How We're Predicting AI - or Failing To

[deleted]
Post reply on HN