EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…
I don’t think anyone familiar with the field is in anyway surprised by this results.
The breakthrough seems really limited to showing it holds for graphs. We already knew this was practically true though anyway.
So... I don't understand. >An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. How do you figure that doesn't allow backprop? A neuronal bit is a loop of neurons. Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Neuron fires dendrite to axon, secondary neuron fires…
> How do you figure that doesn't allow backprop? When you backprop through a linear layer (a matrix W), you need to multiply with W.transpose which is impossible if connections are not bidirectional. > Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Local error aggregation can have a similar effect with backprop but you…
You're conflating the implementation with the principle. There is no matrix math with neurons; quite the opposite, we posit the existence of the matrix math from the behavior we've observed with neural systems governed by a sigmoid function. The equations we've derived are secondary to the initial implementation. Just as you tweak the error factor in backprop, so too do weights between intersecting neuron networks adjust until thought and intention falls into line with eventual perception and execution.
> How do you figure that doesn't allow backprop? When you backprop through a linear layer (a matrix W), you need to multiply with W.transpose which is impossible if connections are not bidirectional. > Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Local error aggregation can have a similar effect with backprop but you…
You're conflating the implementation with the principle. There is no matrix math with neurons; quite the opposite, we posit the existence of the matrix math from the behavior we've observed with neural systems governed by a sigmoid function. The equations we've derived are secondary to the initial implementation. Just as you tweak the error factor in backprop, so too do weights between intersecting neuron networks ad…
> There is no matrix math with neurons
Are we talking about the brain or neural nets biological plausible neural nets?
No. We really have no idea what is going on. We only know some basic psychology about it (holding 7 things in short term, etc.) If we knew something about implementation, we could implement human-like memory.
I suggest starting with the works by Howard Eichenbaum on memory and Edvard & May-Britt Moser (and John O'Keefe and Lynn Nadel) on place & grid cells. For the latest and greatest see https://twitter.com/doellerlab https://twitter.com/KordingLab https://twitter.com/preston_lab https://twitter.com/memorylab https://twitter.com/ptoncompmemlab https://twitter.com/MillerLabMIT https://twitter.com/hugospiers Once you start…
Unless you know of working implementations of memory algorithms I tend to agree that we have no clue how memory works.
So... I don't understand. >An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. How do you figure that doesn't allow backprop? A neuronal bit is a loop of neurons. Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Neuron fires dendrite to axon, secondary neuron fires…
> If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Presumably part of the feedback loop (at least for things like motor skills and rote memorisation) is external to the brain. Our brain causes us to act, which alters our perceptions, which causes the brain to adjust.
Annnd... What exactly encodes that perception thing? The brain! There's nothing magicabout sense data except that it's how we're used to tweaking things. It is still, in the end, just more neurons telling other neurons what's off. Sure, some are hooked up to photoreceptors. I could swap those out with about anything else and still get learning. Unfortunately, the practical benefit of mastery of stuff you imagine tends to be less useful than it feels like it should be IRL.
I suggest starting with the works by Howard Eichenbaum on memory and Edvard & May-Britt Moser (and John O'Keefe and Lynn Nadel) on place & grid cells. For the latest and greatest see https://twitter.com/doellerlab https://twitter.com/KordingLab https://twitter.com/preston_lab https://twitter.com/memorylab https://twitter.com/ptoncompmemlab https://twitter.com/MillerLabMIT https://twitter.com/hugospiers Once you start…
Unless you know of working implementations of memory algorithms I tend to agree that we have no clue how memory works.
Is this approach "more local", in the sense that you could build hardware where local units got work done with less communication? That would have potential. It's feasible to build ICs with a few million simple compute units if they don't have to talk to each other or to memory much. GPUs are a few hundred or a few thousand parallel units that talk to memory a lot.
I suggest starting with the works by Howard Eichenbaum on memory and Edvard & May-Britt Moser (and John O'Keefe and Lynn Nadel) on place & grid cells. For the latest and greatest see https://twitter.com/doellerlab https://twitter.com/KordingLab https://twitter.com/preston_lab https://twitter.com/memorylab https://twitter.com/ptoncompmemlab https://twitter.com/MillerLabMIT https://twitter.com/hugospiers Once you start…
Unless you know of working implementations of memory algorithms I tend to agree that we have no clue how memory works.
The article says this: > The backpropagation algorithm requires information to flow forward and backward along the network. But biological neurons are one-directional. An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. An axon potential never travels backward from a cell's terminals to its body. The point of the research here is that backpropagation turns out…
So... I don't understand. >An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. How do you figure that doesn't allow backprop? A neuronal bit is a loop of neurons. Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Neuron fires dendrite to axon, secondary neuron fires…
>> If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything?
Personally, when I've come to the point where I'm thinking to myself "that must be it, what else can it be?", I am at the point where I need to do more work to answer the latter part of the question.