Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep physical neural networks trained with backpropagation

nature.com

Post reply on HN
Print view View original

11–20 of 38 posts

Re: Deep physical neural networks trained with backpropagation

#11

Post by dekhn »

whatshisface wrote: post #9
pmayrgundter wrote: post #7

Earlier quoted context omitted.

Got me wondering how this compares with neural efficiency, realizing ofc that there's nothing really apples-to-apples here. Training one of these big models takes 100kWh for 1e19 flops, so that's 100k Wh, 360M Ws, or 360MJ or 3.6 1e8J. 1e8Joules/1e19flops = 1e-11J/flop Neurons take 1e-8J/spike.[1] Math check appreciated :) Does seem plausible to think of a single neuron spike (hodgkin-huxley cable model) being modele…

The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.

For those who are curious, consciousness is an epiphenomenon (an emergenty property of brains), while neural spikes are just physics.

See more: https://en.wikipedia.org/wiki/Neural_correlates_of_conscious...

Re: Deep physical neural networks trained with backpropagation

#12

Post by whatshisface »

dekhn wrote: post #11
whatshisface wrote: post #9

Earlier quoted context omitted.

The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.

For those who are curious, consciousness is an epiphenomenon (an emergenty property of brains), while neural spikes are just physics. See more: https://en.wikipedia.org/wiki/Neural_correlates_of_conscious...

I think it would be better to say something like, "paranoia is an epiphenomenon," when nobody knows what consciousness is.

Re: Deep physical neural networks trained with backpropagation

#13

Post by dekhn »

whatshisface wrote: post #12
dekhn wrote: post #11

Earlier quoted context omitted.

For those who are curious, consciousness is an epiphenomenon (an emergenty property of brains), while neural spikes are just physics. See more: https://en.wikipedia.org/wiki/Neural_correlates_of_conscious...

I think it would be better to say something like, "paranoia is an epiphenomenon," when nobody knows what consciousness is.

No. https://en.wikipedia.org/wiki/Epiphenomenalism

Re: Deep physical neural networks trained with backpropagation

#14

Post by netizen-936824 »

whatshisface wrote: post #12
dekhn wrote: post #11

Earlier quoted context omitted.

For those who are curious, consciousness is an epiphenomenon (an emergenty property of brains), while neural spikes are just physics. See more: https://en.wikipedia.org/wiki/Neural_correlates_of_conscious...

I think it would be better to say something like, "paranoia is an epiphenomenon," when nobody knows what consciousness is.

That isn't quite true

https://en.m.wikipedia.org/wiki/Consciousness

Re: Deep physical neural networks trained with backpropagation

#15

Post by dangom »

version_five wrote: post #4

This uses a physical system with controllable parameters to compute a forward pass and > using a differentiable digital model, the gradient of the loss is estimated with respect to the controllable parameters. So e.g. they have a tunable laser that shifts the spectrum of an encoded input based on a set of parameters, and then they update the parameters based on a gradient computed from a digital simulation of the las…

Right,

> Here we introduce a hybrid in situ–in silico algorithm, called physics-aware training, that applies backpropagation to train controllable physical systems. Just as deep learning realizes computations with deep neural networks made from layers of mathematical functions, our approach allows us to train deep physical neural networks made from layers of controllable physical systems, even when the physical layers lack any mathematical isomorphism to conventional artificial neural network layers.

To my naive understanding, and please someone correct me if I'm wrong, the point is that they are not controlling the parameters that compute the NN forward pass directly (hence "no mathematical isomorphism to conventional NNs"), but "hyper-parameters" that guide the physical system to do so. For example, rotation angles of mirrors, or distance between filters, instead of intensity values of light. This leads to the non-linear transformations happening in situ, while simpler transformations in the backprop are still computed in-silico.

Re: Deep physical neural networks trained with backpropagation

#16

Post by whatshisface »

dekhn wrote: post #13
whatshisface wrote: post #12

Earlier quoted context omitted.

I think it would be better to say something like, "paranoia is an epiphenomenon," when nobody knows what consciousness is.

No. https://en.wikipedia.org/wiki/Epiphenomenalism

Note that that title ends in "ism," like "Calvinism," or "Evangelicalism."

Re: Deep physical neural networks trained with backpropagation

#17

Post by davesque »

melissalobos wrote: post #5

> Deep-learning models have become pervasive tools in science and engineering. However, their energy requirements now increasingly limit their scalability.[1] They make this claim first, and cite one source. I haven't heard of this as an issue before. Is there anywhere else I could read more on this? [1] https://arxiv.org/abs/2104.10350

I think they may have provided fewer citations because it felt like a less controversial claim. I think the choice of words was just a bit awkward. To me, it seems like they were asserting that deep learning requires lots of computational resources which is common knowledge. In general, this translates to higher energy requirements.

Re: Deep physical neural networks trained with backpropagation

#18

Post by pmayrgundter »

whatshisface wrote: post #9
pmayrgundter wrote: post #7

Earlier quoted context omitted.

Got me wondering how this compares with neural efficiency, realizing ofc that there's nothing really apples-to-apples here. Training one of these big models takes 100kWh for 1e19 flops, so that's 100k Wh, 360M Ws, or 360MJ or 3.6 1e8J. 1e8Joules/1e19flops = 1e-11J/flop Neurons take 1e-8J/spike.[1] Math check appreciated :) Does seem plausible to think of a single neuron spike (hodgkin-huxley cable model) being modele…

The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.

Sorry, didn't mean it quite like that. It's clear neural spike activity exists as a physical process. I'm suggesting that spiking activity may be an epiphenomena more primary brain functions, i.e. information processing, consciousness, etc..

As far as I know, we're closest to showing information processing in the visual cortex (which is highly linear) and we're still a long way from knowing how it works at a neural level. But maybe someone here can update on this?

But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems. But even if so, that's just "information processing".

I'm personally a fan of electromagnetic theories of consciousness[], where the synaptic activity could be an epiphenomenon of supporting a stand EM field.

[]https://en.wikipedia.org/wiki/Electromagnetic_theories_of_co...

Re: Deep physical neural networks trained with backpropagation

#19

Post by whatshisface »

pmayrgundter wrote: post #18
whatshisface wrote: post #9

Earlier quoted context omitted.

The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.

Sorry, didn't mean it quite like that. It's clear neural spike activity exists as a physical process. I'm suggesting that spiking activity may be an epiphenomena more primary brain functions, i.e. information processing, consciousness, etc.. As far as I know, we're closest to showing information processing in the visual cortex (which is highly linear) and we're still a long way from knowing how it works at a neural l…

>But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems.

I am not sure how much is known about information processing, but it's clear that motor impulses and sensory information are encoded in the spikes. Higher spike frequency = stronger signal. Synapses are how signals are passed from neuron to neuron.

Re: Deep physical neural networks trained with backpropagation

#20

Post by orasis »

phreeza wrote: post #2

Physical/analog computers always suffer from noise limiting their usefulness. So I think it would be natural to apply this to a network architecture that includes noise as an integral Part such as GANs or VAEs.

“noise” is integral to all ML systems. You can view this through many lenses, but generalization can be thought of as decoding a noisy signal.
Post reply on HN