Thermodynamic Natural Gradient Descent
1–10 of 34 posts
Re: Thermodynamic Natural Gradient Descent
#2I know they mainly present results on deep learning/neural network training and optimization, but I wonder how easy it would be to use the same optimization framework for other classes of hard or large optimization problems. I was also curious about this when I saw posts about Extropic (https://www.extropic.ai/) stuff for the first time.
I tried looking into any public info on their website about APIs or software stack to see what's possible beyond NN stuff to model other optimization problems. It looks like that's not shared publicly yet.
There are certainly many NP-hard and large combinatorial or analytical optimization problems still out there that are worth being able to tackle with new technology. Personally, I care about problems in EDA and semiconductor design. Adiabatic quantum computing was one technology with the promise of solving optimization problems (and quantum computing is still playing out with only small-scale solutions at the moment). Hoping that these new "thermodynamic computing" startups also might provide some cool technology to explore these problems with.
Re: Thermodynamic Natural Gradient Descent
#3Sounds great until
> requires an analog thermodynamic computer
Wait. What?
Perhaps a trained physicist can comment on that. Thanks.
Re: Thermodynamic Natural Gradient Descent
#4Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
I believe one example would be quantum annealers. Where "programming" involves setting the right initial conditions and allowing thermodynamics to bring you to an optimum via relaxation.
Re: Thermodynamic Natural Gradient Descent
#5Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
The paper describes it pretty well in appendix C. A matrix of integrators is constructed with a bunch of opamps, RC time constants (using digital potentiometers, presumably) and a multichannel ADC/DAC interface to the PC. Essentially a dedicated differential-equation solver.
So it's a combination of old-school analog computation and modern GPU-based code. Takes longer in practice due to the overhead of interfacing with the hardware and waiting for the integrators to settle, but the authors are claiming that an optimized implementation could outperform a purely-digital solution, as I understand it, by accelerating convergence.
The core idea being that conventional gradient descent is a linear operation at heart, while the gradients actually being traversed are curved surfaces that have to be approximated with multiple unnecessary steps if everything is done in the digital domain.
The trouble, as everybody from Seymour Cray onward has learned the hard way, is that CMOS always wins in the end, simply because the financial power of an entire industry goes into optimizing it.
Re: Thermodynamic Natural Gradient Descent
#6Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
The whole point is to leverage the laws of nature to train AI models, overcoming the limitations and scaling challenges of digital hardware and existing training methods.
Re: Thermodynamic Natural Gradient Descent
#7Cool and interesting. The authors propose a hybrid digital-analog training loop that takes into account the curvature of the loss landscape (i.e., it uses second-order derivatives), and show with numerical simulations that if their method is implemented in a hybrid digital-analog physical system, each iteration in the training loop would incur computational cost that is linear in the number of parameters. I'm all for figuring out ways to let the Laws of Thermodynamics do the work of training AI models, if doing so enables us to overcome the scaling limitations and challenges of existing digital hardware and training methods.
Re: Thermodynamic Natural Gradient Descent
#8Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
The paper describes it pretty well in appendix C. A matrix of integrators is constructed with a bunch of opamps, RC time constants (using digital potentiometers, presumably) and a multichannel ADC/DAC interface to the PC. Essentially a dedicated differential-equation solver. So it's a combination of old-school analog computation and modern GPU-based code. Takes longer in practice due to the overhead of interfacing wi…
I didn't realize they included details about the hardware. Lie you said these just look like analog computers, compute in memory, analog arrays, which have also made a resurgence with deep leaning.
Re: Thermodynamic Natural Gradient Descent
#9Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
This could be attractive if they can build a product along those lines: tens, if not hundreds, of billions of dollars are spent yearly on numerical optimization worldwide, and if this can significantly accelerate it, it could be very profitable.
Re: Thermodynamic Natural Gradient Descent
#10Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
The paper describes it pretty well in appendix C. A matrix of integrators is constructed with a bunch of opamps, RC time constants (using digital potentiometers, presumably) and a multichannel ADC/DAC interface to the PC. Essentially a dedicated differential-equation solver. So it's a combination of old-school analog computation and modern GPU-based code. Takes longer in practice due to the overhead of interfacing wi…
First author of the paper here. That's it indeed! One thing is that this is entirely CMOS-compatible. You could also do something similar with optics or other platforms, but we chose electronic circuits for this reason specifically.