Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thermodynamic Natural Gradient Descent

arxiv.org

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Thermodynamic Natural Gradient Descent

#11

Post by kaelan123 »

stefanpie wrote: post #2

I know they mainly present results on deep learning/neural network training and optimization, but I wonder how easy it would be to use the same optimization framework for other classes of hard or large optimization problems. I was also curious about this when I saw posts about Extropic ( https://www.extropic.ai/ ) stuff for the first time. I tried looking into any public info on their website about APIs or software s…

Indeed, other solving other optimization problem is an interesting avenue. All I can say is stay tuned!

Re: Thermodynamic Natural Gradient Descent

#12

Post by mikewarot »

uoaei wrote: post #4
G3rn0ti wrote: post #3

Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.

I believe one example would be quantum annealers. Where "programming" involves setting the right initial conditions and allowing thermodynamics to bring you to an optimum via relaxation.

Dwave[1] does compute that way already.

[1] https://www.dwavesys.com/

Re: Thermodynamic Natural Gradient Descent

#13

Post by jasonjmcghee »

G3rn0ti wrote: post #3

Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.

From my understanding, this is exactly what https://extropic.ai is working on, and I wouldn't be surprised if https://normalcomputing.ai/ (authors of the paper) is as well.

Re: Thermodynamic Natural Gradient Descent

#14

Post by thomasahle »

The main point of this is that natural gradient descent is a second-order method. The main GD update equation is:

∇̃L(θ) = F⁻¹∇L(θ)

which requires solving a linear system. For this, you can use the methods from the author's previous paper [Thermodynamic Linear Algebra](https://arxiv.org/abs/2308.05660).

Since it's hard to implement a full neural network on a thermodynamic computer, the paper suggests running one in parallel to a normal GPU. The GPU computes F and ∇L(θ), but offloads the linear system to the thermo computer, which runs in parallel to the digital system (Figure 1).

It is important to note that the "Runtime vs Accuracy" plot in Figure 3 uses a "timing model" for the TNGD algorithm, since the computer necessary to run the algorithm still doesn't exist.

Re: Thermodynamic Natural Gradient Descent

#15

Post by kaelan123 »

mikewarot wrote: post #12
uoaei wrote: post #4

Earlier quoted context omitted.

I believe one example would be quantum annealers. Where "programming" involves setting the right initial conditions and allowing thermodynamics to bring you to an optimum via relaxation.

Dwave[1] does compute that way already. [1] https://www.dwavesys.com/

One key difference is the system is entirely classical (not quantum) and noise-resilient (see the last appendix).

Re: Thermodynamic Natural Gradient Descent

#16

Post by CamperBob2 »

kaelan123 wrote: post #10
CamperBob2 wrote: post #5

Earlier quoted context omitted.

The paper describes it pretty well in appendix C. A matrix of integrators is constructed with a bunch of opamps, RC time constants (using digital potentiometers, presumably) and a multichannel ADC/DAC interface to the PC. Essentially a dedicated differential-equation solver. So it's a combination of old-school analog computation and modern GPU-based code. Takes longer in practice due to the overhead of interfacing wi…

First author of the paper here. That's it indeed! One thing is that this is entirely CMOS-compatible. You could also do something similar with optics or other platforms, but we chose electronic circuits for this reason specifically.

By that remark I meant "digital CMOS," in the sense of elements that store state information discretely in flip-flops or gate insulators rather than continuously with analog integrators.

Very cool work in any event, though! Best of luck with the ongoing R&D.

Re: Thermodynamic Natural Gradient Descent

#17

Post by stefanpie »

kaelan123 wrote: post #11
stefanpie wrote: post #2

I know they mainly present results on deep learning/neural network training and optimization, but I wonder how easy it would be to use the same optimization framework for other classes of hard or large optimization problems. I was also curious about this when I saw posts about Extropic ( https://www.extropic.ai/ ) stuff for the first time. I tried looking into any public info on their website about APIs or software s…

Indeed, other solving other optimization problem is an interesting avenue. All I can say is stay tuned!

Hey thanks for the reply! I'm assuming your the first author on the paper; if so, the signup button on the Normal Computing website is not working at the moment (at least for me, even with ad blocker turned off).

Re: Thermodynamic Natural Gradient Descent

#18

Post by nickpsecurity »

G3rn0ti wrote: post #3

Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.

Analog computers have a lot of history. You can Google analog with neural network or differential equations to get many results. They are fast with low power, can have precision issues, and require custom, chip design.

https://en.m.wikipedia.org/wiki/Analog_computer

Mixed signal ASIC’s often use a mix of digital and analog blocks to get the benefits of analog. It’s especially helpful for anything that eats lots of power or to prevent that (eg mobile).

Re: Thermodynamic Natural Gradient Descent

#19

Post by esafak »

Not having read the paper carefully, could someone tell me what the draw is? It looks like it is going to have the same asymptotic complexity as SGD in terms of sample size, per Table 1. Given that today's large, over-specified models have numerous, comparable extrema, is there even a need for this? I wouldn't get out of bed unless it were sublinear.
Post reply on HN