Deep physical neural networks trained with backpropagation
1–10 of 38 posts
Re: Deep physical neural networks trained with backpropagation
#2Physical/analog computers always suffer from noise limiting their usefulness. So I think it would be natural to apply this to a network architecture that includes noise as an integral Part such as GANs or VAEs.
Re: Deep physical neural networks trained with backpropagation
#3How is this different from the good old “chip in the loop” training method?
Re: Deep physical neural networks trained with backpropagation
#4This uses a physical system with controllable parameters to compute a forward pass and
> using a differentiable digital model, the gradient of the loss is estimated with respect to the controllable parameters.
So e.g. they have a tunable laser that shifts the spectrum of an encoded input based on a set of parameters, and then they update the parameters based on a gradient computed from a digital simulation of the laser (physics aware model).
When I read the headline I imagined they had implemented back propagation in a physical system
Re: Deep physical neural networks trained with backpropagation
#5> Deep-learning models have become pervasive tools in science and engineering. However, their energy requirements now increasingly limit their scalability.[1]
They make this claim first, and cite one source. I haven't heard of this as an issue before. Is there anywhere else I could read more on this?
Re: Deep physical neural networks trained with backpropagation
#6> Deep-learning models have become pervasive tools in science and engineering. However, their energy requirements now increasingly limit their scalability.[1] They make this claim first, and cite one source. I haven't heard of this as an issue before. Is there anywhere else I could read more on this? [1] https://arxiv.org/abs/2104.10350
I don't have a specific reference but I'd say it's a common knowledge assertion based on the growth in the number of parameters in models over the last 10 years. There are lots of places where you can see how the number of parameters, especially in language and vision models, has increased, and find that the amount of training time quoted. Normally it's framed in terms of compute instead of energy.
Re: Deep physical neural networks trained with backpropagation
#7> Deep-learning models have become pervasive tools in science and engineering. However, their energy requirements now increasingly limit their scalability.[1] They make this claim first, and cite one source. I haven't heard of this as an issue before. Is there anywhere else I could read more on this? [1] https://arxiv.org/abs/2104.10350
Got me wondering how this compares with neural efficiency, realizing ofc that there's nothing really apples-to-apples here.
Training one of these big models takes 100kWh for 1e19 flops, so that's 100k Wh, 360M Ws, or 360MJ or 3.6 1e8J. 1e8Joules/1e19flops = 1e-11J/flop
Neurons take 1e-8J/spike.[1]
Math check appreciated :)
Does seem plausible to think of a single neuron spike (hodgkin-huxley cable model) being modeled with ~1k flops. Though I'm firmly of the opinion that nobody really knows how the brain works.. the neural spike activity could be pure epiphenomenon.. who knows!
[1] “Finally, the energy supply to a neuron by ATP is 8.31 × 10−9 J. Meanwhile, integrating the total power with respect to time we will get the consumed electric power, which is 8.75 × 10−9 J. This is more energy than the ATP supplied. The energy efficiency is 105.3%. This is an anomaly…” - 2017 Feb 16 Wang, Xu, Institute for Cognitive Neurodynamics, East China University of Science and Technology https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5337805/
Re: Deep physical neural networks trained with backpropagation
#8How is this different from the good old “chip in the loop” training method?
The paper is interesting
Re: Deep physical neural networks trained with backpropagation
#9> Deep-learning models have become pervasive tools in science and engineering. However, their energy requirements now increasingly limit their scalability.[1] They make this claim first, and cite one source. I haven't heard of this as an issue before. Is there anywhere else I could read more on this? [1] https://arxiv.org/abs/2104.10350
Got me wondering how this compares with neural efficiency, realizing ofc that there's nothing really apples-to-apples here. Training one of these big models takes 100kWh for 1e19 flops, so that's 100k Wh, 360M Ws, or 360MJ or 3.6 1e8J. 1e8Joules/1e19flops = 1e-11J/flop Neurons take 1e-8J/spike.[1] Math check appreciated :) Does seem plausible to think of a single neuron spike (hodgkin-huxley cable model) being modele…
The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.
Re: Deep physical neural networks trained with backpropagation
#10> Deep-learning models have become pervasive tools in science and engineering. However, their energy requirements now increasingly limit their scalability.[1] They make this claim first, and cite one source. I haven't heard of this as an issue before. Is there anywhere else I could read more on this? [1] https://arxiv.org/abs/2104.10350
Training a state of the art model typically involves keeping a very large computer around at near 100% power load. Roughly about 10MW.
The actual limits on DL models (and any simulation or optimization) are: power density and the speed of light, plus the maximum amount of power you can deliver to the area. The speed of light limits how long your cables can be while still doing collective reductions, and the power density limits how much compute power you can fit per unit volume. One could imagine a fully liquid cooled supercomputer at 100MW (located near a very reliable and large power source) with optical fiber interconnect, this would completely change the state of the art in large models overnight.