Training of Physical Neural Networks • Hacker News Board

Re: Training of Physical Neural Networks

#21

Post by tomxor » Wed, Jul 10, 2024, 8:03 PM UTC

immibis wrote: post #18

tomxor wrote: post #17

Earlier quoted context omitted.

Some parts are copiable, but not the more abstract things like the human intellect, for lack of a better word. We are not even born with what you might consider basic mental faculties, for example it might seem absurd, but we have to learn to see... We are born with the "hardware" for it, a visual cortex, an eye, all defined by our genes, but it's actually trained from birth, there is even a feedback loop that causes…

They raised some cats from birth in an environment with only vertically-oriented edges, none horizontal. Those cats could not see horizontally-oriented things. https://computervisionblog.wordpress.com/2013/06/01/cats-and... Likewise, kittens with an eye patch over an eye in the same time period remain blind in that eye forever.

Wow, that's a horrific way of proving that theory.

Re: Training of Physical Neural Networks

#22

Post by tomxor » Wed, Jul 10, 2024, 8:06 PM UTC

TheLoafOfBread wrote: post #20

tomxor wrote: post #10

Earlier quoted context omitted.

ANN is taken.

I mean LoRA was taken too before LoRA became a thing

I don't mean globally, LoRA are at least in different domains. Artificial Neural Networks and Physical Neural Networks are both machine learning, discussion referring to both is highly probable, and the former far more established so it calling it an Analog Neural Network would never last long.

Re: Training of Physical Neural Networks

#23

Post by alexpotato » Wed, Jul 10, 2024, 9:41 PM UTC

tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

> Last time I read about this the main practical difficulty was model transferability.

There is a great write up of this in this old blog post: https://www.damninteresting.com/on-the-origin-of-circuits/

Re: Training of Physical Neural Networks

#24

Post by alexpotato » Wed, Jul 10, 2024, 9:42 PM UTC

tomxor wrote: post #17

trextrex wrote: post #16

Earlier quoted context omitted.

Well, the brain is a physical neural network, and evolution seems to have figured out how to generate a (somewhat) copiable model. I bet we could learn a trick or two from biology here.

Some parts are copiable, but not the more abstract things like the human intellect, for lack of a better word. We are not even born with what you might consider basic mental faculties, for example it might seem absurd, but we have to learn to see... We are born with the "hardware" for it, a visual cortex, an eye, all defined by our genes, but it's actually trained from birth, there is even a feedback loop that causes…

Another example:

Children who were "raised in the wild" or locked in a room by themselves have shown to be incapable of learning full human language.

The working theory is that our brains can only learn certain skills at certain times of brain development/ages.

Re: Training of Physical Neural Networks

#25

Post by robertsdionne » Wed, Jul 10, 2024, 9:43 PM UTC

tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

This is “Mortal Computation” coined in Hinton’s The Forward-Forward Algorithm: Some Preliminary Investigations https://arxiv.org/abs/2212.13345.

Re: Training of Physical Neural Networks

#26

Post by BriggyDwiggs42 » Wed, Jul 10, 2024, 10:39 PM UTC

immibis wrote: post #18

tomxor wrote: post #17

Earlier quoted context omitted.

Some parts are copiable, but not the more abstract things like the human intellect, for lack of a better word. We are not even born with what you might consider basic mental faculties, for example it might seem absurd, but we have to learn to see... We are born with the "hardware" for it, a visual cortex, an eye, all defined by our genes, but it's actually trained from birth, there is even a feedback loop that causes…

They raised some cats from birth in an environment with only vertically-oriented edges, none horizontal. Those cats could not see horizontally-oriented things. https://computervisionblog.wordpress.com/2013/06/01/cats-and... Likewise, kittens with an eye patch over an eye in the same time period remain blind in that eye forever.

Geez poor kitties, but that is interesting.

Re: Training of Physical Neural Networks

#27

Post by programjames » Wed, Jul 10, 2024, 11:25 PM UTC

tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

You can regularize the networks to make them transfer easier. I can't remember the abstract's title off the top of my head though.

Re: Training of Physical Neural Networks

#28

Post by programjames » Wed, Jul 10, 2024, 11:26 PM UTC

> These methods are typically slow because the number of gradient updates scales linearly with the number of learnable parameters in the network, posing a significant challenge for scaling up.

This is a pretty big problem, though if you use information-bottleneck training you can train each layer simultaneously.

Re: Training of Physical Neural Networks

#29

Post by eightysixfour » Wed, Jul 10, 2024, 11:26 PM UTC

ksd482 wrote: post #4

PNNs resemble neural networks, however at least part of the system is analog rather than digital, meaning that part or all the input/output data is encoded continuously in a physical parameter, and the weights can also be physical, with the ultimate goal of surpassing digital hardware in performance or efficiency. I am trying to understand what format does a node take in PNNs. Is it a transistor? Or is it more comple…

Here you go: https://arstechnica.com/science/2018/07/neural-network-imple...

Re: Training of Physical Neural Networks

#30

Post by deepfriedchokes » Wed, Jul 10, 2024, 11:27 PM UTC

alexpotato wrote: post #24

tomxor wrote: post #17

Earlier quoted context omitted.

Some parts are copiable, but not the more abstract things like the human intellect, for lack of a better word. We are not even born with what you might consider basic mental faculties, for example it might seem absurd, but we have to learn to see... We are born with the "hardware" for it, a visual cortex, an eye, all defined by our genes, but it's actually trained from birth, there is even a feedback loop that causes…

Another example: Children who were "raised in the wild" or locked in a room by themselves have shown to be incapable of learning full human language. The working theory is that our brains can only learn certain skills at certain times of brain development/ages.

We should also consider the effects of trauma on those brains. If you’ve ever spent time around people with extreme trauma they are very much in their own heads and can’t focus outside themselves long enough to focus enough to learn anything. It definitely impacts intellectual capacity. Humans are social animals and anyone raised without proper socializing and intimacy and nurturing will inevitably end up traumatized.