Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training of Physical Neural Networks

arxiv.org

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Training of Physical Neural Networks

#41

Post by etiam »

dsabanin wrote: post #33
tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

Couldn’t you still copy by training a new network on a new device to have same outputs for the same inputs as the original?

Distillation (as you may be aware). https://arxiv.org/abs/1503.02531

Having to do that in each instance is still really cumbersome for cheap mass deployment compared to just making a digital-style exact copy, but then again I guess a main argument for wanting these systems is that they'd be doing things unachievable in practice on digital computers.

In some cases one might be able to distill to digital arithmetic after the heavy parts of the optimization are done, for replication, distribution, better access for software analysis, etc.

Re: Training of Physical Neural Networks

#42

Post by actionfromafar »

bongodongobob wrote: post #12
tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

Reminds me of the evolutionary FPGA experiment that was dependent on magnetic flux or something. The same program wouldn't work on a different FPGA.

Would be interesting to hook up many FPGAs of the same model and train all of the at once. Programs with differing outputs on different individuals could be discarded. The program may still not transfer to another batch of FPGAs but at least you have a better chance of the working.

Another idea is to just train a whole bunch of them individually, like putting your chips in school. :-D

Re: Training of Physical Neural Networks

#43

Post by CuriouslyC »

tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

This was the thing Geoff Hinton cited as a problem with analog networks.

I think eventually we'll get to the point where we do a stage of pretraining on noisy digital hardware to create a transferrable network, then fine tune it on the analog system.

Re: Training of Physical Neural Networks

#44

Post by jegp »

tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

It's still possible to train a network that's aware of the physics and then transfer that to physical devices. One approach to this from the neuromorphic community (that's been working on this for a long time) is called the Neuromorphic Intermediate Representation (NIR) and already lets you transfer models to several hardware platforms [1]. This is pretty cool because we can use the same model across systems, similar to a digital instruction set. Ofc, this doesn't fix the problem of sensitivity. But biology fixed that with plasticity, so we can probably learn to circumvent that.

[1]: https://github.com/neuromorphs/nir (disclaimer: I'm one of the authors)

Re: Training of Physical Neural Networks

#46

Post by orbifold »

Shawnecy wrote: post #7

My knowledge in this area is incredibly limited, but I figured the paper would mention NanoWire Networks (NWNs) as an emerging physical neural network[0]. Last year, researchers from the University of Sydney and UCLA used NWNs to demonstrate online learning of handwritten digits with an accuracy of 93%. [0] = https://www.nature.com/articles/s41467-023-42470-5

Classifying MNIST digits with 93% accuracy can also be accomplished using a linear classifier. So it isn't clear to me what the advantage would be.

Re: Training of Physical Neural Networks

#47

Post by 6gvONxR4sf7o »

tomxor wrote: post #9

Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…

If (somehow/waves hands) you could parallelize training, maybe this would turn into an implicit regularization and be a benefit, not a flaw. Then again, physical parallelizability might be an infeasibly restrictive constraint?

Re: Training of Physical Neural Networks

#48

Post by l33tman »

tomxor wrote: post #34
dsabanin wrote: post #33

Earlier quoted context omitted.

Couldn’t you still copy by training a new network on a new device to have same outputs for the same inputs as the original?

Yes, but training is the most expensive part of ML, for example GPT-3 is estimated to cost something like 1-4 million USD. With ANN you can do it one time and then clone the result for negligible energy cost. Maybe training a batch of PNNs in parallel could save some of the energy cost, but I don't know how feasible that is considering they could behave slightly differently during training causing divergence... Now t…

That's not true for the most well-known models. For example Meta's LLAMA training and architecture was predicated on the observation that training cost is a drop in the well compared to the inference cost for a model's lifetime.

Re: Training of Physical Neural Networks

#49

Post by dcuthbertson »

lynx23 wrote: post #38
hansworst wrote: post #19

Earlier quoted context omitted.

The way the brain does it is by giving users a largely untrained model that they themselves have to train over the next 20 years for it to be of any use.

20 years of training is not enough. Neuroscientists say 25. According to my own experience, its more like 30.

In the end, it's a life-long process.

Re: Training of Physical Neural Networks

#50

Post by tomxor »

wbillingsley wrote: post #39
hansworst wrote: post #19

Earlier quoted context omitted.

The way the brain does it is by giving users a largely untrained model that they themselves have to train over the next 20 years for it to be of any use.

Sometimes. Foals are born (almost) able to walk. There are occasions where evolution baked the model into the genes.

Yeah that example came to my mind too.

I suspect there may be trade off undergoing evolutionary selection here, where for some organisms a behaviour is more important from the offset, it's worth encoding more of the behaviour into genes, at what cost I wonder?

It's also possible there is some other mechanism going on at an embryonic stage, a kind of pre-training.

I suspect some of the division is also defined by how complex the task is, or how sensitive the model is to it's own neurons (kind of like PNN). I don't have a well rounded argument, but my instinct is that encoding or pre-training walking is far easier than seeing. Not to mention basic quadrupedal walking/standing is far easier than bipedal, they can learn the more complex coordinated movements after.

Post reply on HN