Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists watch a memory form in the brain of a living fish

quantamagazine.org

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Scientists watch a memory form in the brain of a living fish

#21

Post by igorkraw »

sva_ wrote: post #16
rakkhi wrote: post #5

In fish... "Contrary to expectation, the synaptic strengths in the pallium remained about the same regardless of whether the fish learned anything. Instead, in the fish that learned, the synapses were pruned from some areas of the pallium — producing an effect “like cutting a bonsai tree,” Fraser said — and replanted in others." This is a very counter intuitive. So there are existing neural connections (formed someho…

I don't think it is counter intuitive (maybe if you hold a strong opinion, it is.) There's a growing body of evidence that brains, to some degree, learn by removing connections, and in some cases, add connections elsewhere. I think part of learning is, that we have some general idea about something, but it turns out to be inaccurate, so we need to specialize it (and get rid off the old connection.) I'm speculating th…

Artificial neural networks have something similar, called the lottery ticket hypothesis

Re: Scientists watch a memory form in the brain of a living fish

#22

Post by kenjackson »

rakkhi wrote: post #5

In fish... "Contrary to expectation, the synaptic strengths in the pallium remained about the same regardless of whether the fish learned anything. Instead, in the fish that learned, the synapses were pruned from some areas of the pallium — producing an effect “like cutting a bonsai tree,” Fraser said — and replanted in others." This is a very counter intuitive. So there are existing neural connections (formed someho…

I wonder if this means that learning useless information is extra counterproductive in that it may make it difficult to learn other things because you’ve pruned some neutral connections?

Re: Scientists watch a memory form in the brain of a living fish

#23

Post by agumonkey »

rakkhi wrote: post #5

In fish... "Contrary to expectation, the synaptic strengths in the pallium remained about the same regardless of whether the fish learned anything. Instead, in the fish that learned, the synapses were pruned from some areas of the pallium — producing an effect “like cutting a bonsai tree,” Fraser said — and replanted in others." This is a very counter intuitive. So there are existing neural connections (formed someho…

Maybe instead of adding "new knowledge" our brain works by "removing bad candidates".

Re: Scientists watch a memory form in the brain of a living fish

#24

Post by hashimotonomora »

rakkhi wrote: post #5

In fish... "Contrary to expectation, the synaptic strengths in the pallium remained about the same regardless of whether the fish learned anything. Instead, in the fish that learned, the synapses were pruned from some areas of the pallium — producing an effect “like cutting a bonsai tree,” Fraser said — and replanted in others." This is a very counter intuitive. So there are existing neural connections (formed someho…

That’s counter intuitive (at least for you) because in modern society it’s customary to think that people are born tabula rasa and then “it’s all a social construct”. Perhaps we are indeed born with innate preferences, biases, and default sexual orientations …

Re: Scientists watch a memory form in the brain of a living fish

#26

Post by sva_ »

igorkraw wrote: post #21
sva_ wrote: post #16

Earlier quoted context omitted.

I don't think it is counter intuitive (maybe if you hold a strong opinion, it is.) There's a growing body of evidence that brains, to some degree, learn by removing connections, and in some cases, add connections elsewhere. I think part of learning is, that we have some general idea about something, but it turns out to be inaccurate, so we need to specialize it (and get rid off the old connection.) I'm speculating th…

Artificial neural networks have something similar, called the lottery ticket hypothesis

I'm not sure it is quite comparable, as the lottery hypothesis postulates that for every Neural Net with n connections/weights, there is one with n-1 connections, that retains its accuracy. So it doesn't learn by pruning, just becomes more efficient by being smaller. That's at least how I understood it.

Re: Scientists watch a memory form in the brain of a living fish

#27

Post by dmarchand90 »

hashimotonomora wrote: post #24
rakkhi wrote: post #5

In fish... "Contrary to expectation, the synaptic strengths in the pallium remained about the same regardless of whether the fish learned anything. Instead, in the fish that learned, the synapses were pruned from some areas of the pallium — producing an effect “like cutting a bonsai tree,” Fraser said — and replanted in others." This is a very counter intuitive. So there are existing neural connections (formed someho…

That’s counter intuitive (at least for you) because in modern society it’s customary to think that people are born tabula rasa and then “it’s all a social construct”. Perhaps we are indeed born with innate preferences, biases, and default sexual orientations …

Makes me think of some ancient philosophical theories that all learning is the uncovering of latent memory

Re: Scientists watch a memory form in the brain of a living fish

#28

Post by igorkraw »

sva_ wrote: post #26
igorkraw wrote: post #21

Earlier quoted context omitted.

Artificial neural networks have something similar, called the lottery ticket hypothesis

I'm not sure it is quite comparable, as the lottery hypothesis postulates that for every Neural Net with n connections/weights, there is one with n-1 connections, that retains its accuracy. So it doesn't learn by pruning, just becomes more efficient by being smaller. That's at least how I understood it.

Your thing would imply a 0 connected neural net could compete with GPT3. I quote

The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialised such that — when trained in isolation — it can match the test accuracy of the original network after training for at most the same number of iterations. - Frankle & Carbin (2019, p.2)

Re: Scientists watch a memory form in the brain of a living fish

#29

Post by sva_ »

igorkraw wrote: post #28
sva_ wrote: post #26

Earlier quoted context omitted.

I'm not sure it is quite comparable, as the lottery hypothesis postulates that for every Neural Net with n connections/weights, there is one with n-1 connections, that retains its accuracy. So it doesn't learn by pruning, just becomes more efficient by being smaller. That's at least how I understood it.

Your thing would imply a 0 connected neural net could compete with GPT3. I quote The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialised such that — when trained in isolation — it can match the test accuracy of the original network after training for at most the same number of iterations. - Frankle & Carbin (2019, p.2)

Yeah, that's why it is merely a hypothesis. I think the Lottery Ticket hypothesis goes further and states that this subnetwork also has a subnetwork which matches test accuracy, up to some degree. Else it wouldn't be interesting. Of course there must be a limit...

Re: Scientists watch a memory form in the brain of a living fish

#30

Post by igorkraw »

sva_ wrote: post #29
igorkraw wrote: post #28

Earlier quoted context omitted.

Your thing would imply a 0 connected neural net could compete with GPT3. I quote The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialised such that — when trained in isolation — it can match the test accuracy of the original network after training for at most the same number of iterations. - Frankle & Carbin (2019, p.2)

Yeah, that's why it is merely a hypothesis. I think the Lottery Ticket hypothesis goes further and states that this subnetwork also has a subnetwork which matches test accuracy, up to some degree. Else it wouldn't be interesting. Of course there must be a limit...

The interesting bit is that you don't need to tweak the subnet, only remove the other noisy connections. If you remove capacity, at some point test accuracy needs to go down. The hypothesis is about the idea that we maybe don't "train" the weights, we "find" them, not about unlimited nesting (as far as I understand it, but I have high confidence. Happy to be proven wrong tho)
Post reply on HN