Artificial Neural Nets Finally Yield Clues to How Brains Learn • Hacker News Board

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#21

Post by amirkdv » Sat, Feb 20, 2021, 6:53 PM UTC

NalNezumi wrote: post #15

There's three things I've always been baffled by the lack of interest in the current deep learning based AI field when it comes to parallels with biological brain: 1. Biological plausibility of back prop. 2. The lack of interest/consideration of time-continuous input on network. They are currently discrete and "learning" and inference is done separately. That's not how most organisms work. 3. The lack of consideratio…

You're describing the deep dissonance I was feeling a decade ago when I first stepped into AI research. I just kind of always assumed that studying AI would necessarily have a strong focus on how biological intelligence works. And boy was I wrong.

Knowing a bit more now, this gap makes some sense:

1. Neuroscience is really, really hard. Even with the unbelievable recent advances, we're still years away from having a clear understanding of the mechanics of learning and memory.

2. The drift between AI and the broader cognitive sciences started in the 70s, seemingly borne out of pragmatism and the difference in goals between engineer types and scientist types.

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#22

Post by dualthro » Sat, Feb 20, 2021, 7:04 PM UTC

The_rationalist wrote: post #7

specialist wrote: post #2

With articles like this, I want a "check back in 2 years" reminder, to see how the science shakes out. I'm not smart or informed enough to judge these current events style updates for myself.

Reddit has the remindMe bot for that, HN should give us an exobrain too

You don't think setting a reminder in your calendar for 2 years from now would suffice?

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#23

Post by vmception » Sat, Feb 20, 2021, 7:12 PM UTC

Does anyone else notice that a lot of this stuff is just rehashed forms of things from decades prior?

Someone tried making a computer like this decades ago.

Ex-Machina had a plot device like this too, to make the robot’s transistor based brain.

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#24

Post by x1798DE » Sat, Feb 20, 2021, 7:14 PM UTC

visarga wrote: post #13

specialist wrote: post #2

With articles like this, I want a "check back in 2 years" reminder, to see how the science shakes out. I'm not smart or informed enough to judge these current events style updates for myself.

Check back the predictions of 2 years ago and compare to the reality of today.

I occasionally did check up on stories, but people rarely do follow-up reporting (especially for things that don't pan out), and Google searches usually just turn up 50 variations on the original story written from the original press release. It's a very unfortunate dynamic.

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#25

Post by rantwasp » Sat, Feb 20, 2021, 7:17 PM UTC

NalNezumi wrote: post #15

There's three things I've always been baffled by the lack of interest in the current deep learning based AI field when it comes to parallels with biological brain: 1. Biological plausibility of back prop. 2. The lack of interest/consideration of time-continuous input on network. They are currently discrete and "learning" and inference is done separately. That's not how most organisms work. 3. The lack of consideratio…

not an expert (more like a noob) by any means but:

1) from neuroscience point of view you have cortical columns with layers that are wired to send the input forward but to also propagate feedback. the layers constantly predict what is going to happen (by having neurons fire) and usually it’s the delta between what is predicted and what is coming from the sensory system that drives the reinforcement or the weakening of the connections. this sort of sounds like backpropagation to me (but again i may be super ignorant and would appreciate if you can educate me on this if you know more)

2) technically the “input” in the brain is not continuous. I don’t want to go into semantics but at the end of the day you have molecules, ions etc. so the input/transmission is not continuous. the size of the neurotransmitters is so small that it looks like it’s continuous. my point is that, if you take the current model and you have more computing power you could find out that some things translate between the 2 models (we definitely need a way better model of the neuron, but that’s another story)

3) this is a fair point.

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#26

Post by canjobear » Sat, Feb 20, 2021, 7:41 PM UTC

NalNezumi wrote: post #15

There's three things I've always been baffled by the lack of interest in the current deep learning based AI field when it comes to parallels with biological brain: 1. Biological plausibility of back prop. 2. The lack of interest/consideration of time-continuous input on network. They are currently discrete and "learning" and inference is done separately. That's not how most organisms work. 3. The lack of consideratio…

Backprop isn't biologically plausible but predictive coding is and it approximates backprop.

https://arxiv.org/abs/2006.04182

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#27

Post by The_rationalist » Sat, Feb 20, 2021, 8:09 PM UTC

dualthro wrote: post #22

The_rationalist wrote: post #7

Earlier quoted context omitted.

Reddit has the remindMe bot for that, HN should give us an exobrain too

You don't think setting a reminder in your calendar for 2 years from now would suffice?

It's too much clicks away, it's should be a matter of one click

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#28

Post by mam2 » Sat, Feb 20, 2021, 8:21 PM UTC

NalNezumi wrote: post #15

There's three things I've always been baffled by the lack of interest in the current deep learning based AI field when it comes to parallels with biological brain: 1. Biological plausibility of back prop. 2. The lack of interest/consideration of time-continuous input on network. They are currently discrete and "learning" and inference is done separately. That's not how most organisms work. 3. The lack of consideratio…

1 backprop is not necessarily the only thing able to perform optim.. it could be something more parallel that try many path at once. a bit like quantum computing.. but we have not just found the algo yet

2 is basically sleeping.

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#29

Post by FL33TW00D » Sat, Feb 20, 2021, 9:09 PM UTC

NalNezumi wrote: post #15

There's three things I've always been baffled by the lack of interest in the current deep learning based AI field when it comes to parallels with biological brain: 1. Biological plausibility of back prop. 2. The lack of interest/consideration of time-continuous input on network. They are currently discrete and "learning" and inference is done separately. That's not how most organisms work. 3. The lack of consideratio…

The optimist in me likes to liken it to the difference between birds and planes. Same result but different principle.

Re: Artificial Neural Nets Finally Yield Clues to How Brains Learn

#30

Post by sdenton4 » Sat, Feb 20, 2021, 11:18 PM UTC

NalNezumi wrote: post #15

There's three things I've always been baffled by the lack of interest in the current deep learning based AI field when it comes to parallels with biological brain: 1. Biological plausibility of back prop. 2. The lack of interest/consideration of time-continuous input on network. They are currently discrete and "learning" and inference is done separately. That's not how most organisms work. 3. The lack of consideratio…

For 2, it's worth looking at streaming audio input; we often use rnn's, which update state continually over time, and can make multi scale representations of the audio to help guide decisions. (eg, speaker embeddings which apply to whole sentences, and guide sample-level speech separation.). Audio's a great place to experiment with time; it's much cheaper than dealing with video, and there are lots of great data sets.

Transformers and attention are also tools for responding to the current context with a pretrained network.

A core problem is that if you update the weights during inference, there's seemingly very little garauntee of keeping prior quality high, and the models are already occult magic as it is. Federated learning might be another interesting area to look into for ways to address that issue.