this reminds me of simulated annealing which I learned about in an AI class about a decade ago.
Thermodynamic Natural Gradient Descent
21–30 of 34 posts
Re: Thermodynamic Natural Gradient Descent
#22Leveraging thermodynamics to more efficiently compute second-order updates is certainly cool and worth exploring, however specifically in the context of deep learning I remain skeptical of its usefulness.
We already have very efficient second-order methods running on classical hardware [1] but they are basically not being used at all in practice, as they are outperformed by ADAM and other 1st-order methods. This is because optimizing highly nonlinear loss functions, such as the ones in deep learning models, only really works with very low learning rates, regardless of whether a 1st or a 2nd order method is used. So, comparatively speaking, a 2nd order method might give you a slightly better parameter update per step but at a more-than-slightly-higher cost, so most of the time it's simply not worth doing.
[1] https://andrew.gibiansky.com/blog/machine-learning/hessian-f...
Re: Thermodynamic Natural Gradient Descent
#23Leveraging thermodynamics to more efficiently compute second-order updates is certainly cool and worth exploring, however specifically in the context of deep learning I remain skeptical of its usefulness. We already have very efficient second-order methods running on classical hardware [1] but they are basically not being used at all in practice, as they are outperformed by ADAM and other 1st-order methods. This is b…
Agreed that it's very cool, and also about how hard it is to make second order methods worthwhile. We're just using such huge datasets that sometimes it's hard to even get a decent estimate of the gradient for a minibatch. Getting a useful estimate of second order information over the dataset is even harder, especially when the whole point of using minibatches is computational feasibility.
Re: Thermodynamic Natural Gradient Descent
#24Leveraging thermodynamics to more efficiently compute second-order updates is certainly cool and worth exploring, however specifically in the context of deep learning I remain skeptical of its usefulness. We already have very efficient second-order methods running on classical hardware [1] but they are basically not being used at all in practice, as they are outperformed by ADAM and other 1st-order methods. This is b…
Agreed that it's very cool, and also about how hard it is to make second order methods worthwhile. We're just using such huge datasets that sometimes it's hard to even get a decent estimate of the gradient for a minibatch. Getting a useful estimate of second order information over the dataset is even harder, especially when the whole point of using minibatches is computational feasibility.
Those are valid points! Hessian-free (HF) optimization is a really nice method, but as you say remains costly so people don't use it. The key idea in this paper is that if you are able to solve linear systems faster by using an analog device, the cost of a HF-like method is brought down, so the method can become competitive.
About the noise, it is true that the second-order information will be noisier than the gradient for a given batch size (and a lot of results out there for HF optimization are with impractically large batch sizes). In the paper we use relatively small batch sizes (eg 32 for the fine-tuning example) and show that you can still get an advantage from second-order information. Of course it would be interesting to study in more detail how noisy 2nd order information can be, and on more datasets.
Re: Thermodynamic Natural Gradient Descent
#25The main point of this is that natural gradient descent is a second-order method. The main GD update equation is: ∇̃L(θ) = F⁻¹∇L(θ) which requires solving a linear system. For this, you can use the methods from the author's previous paper [Thermodynamic Linear Algebra]( https://arxiv.org/abs/2308.05660 ). Since it's hard to implement a full neural network on a thermodynamic computer, the paper suggests running one in…
It really gives nice way to think about gradient descent.
Re: Thermodynamic Natural Gradient Descent
#26Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
Analog computers have a lot of history. You can Google analog with neural network or differential equations to get many results. They are fast with low power, can have precision issues, and require custom, chip design. https://en.m.wikipedia.org/wiki/Analog_computer Mixed signal ASIC’s often use a mix of digital and analog blocks to get the benefits of analog. It’s especially helpful for anything that eats lots of po…
Hard to beat the string algorithm for finding shortest paths on a positive weights network (e.g. build the network out of string where topologies match and link lengths are link weights, find the origin and destination nodes/knots of interest, grab the two nodes and pull until taut).
Or the spaghetti approach to finding the largest value from a list of positive values (e.g. cut dry spaghetti noodles to length for each value, bundle them together and tap the bundle on a table, the one visually sticking out the most is the largest valued element).
Re: Thermodynamic Natural Gradient Descent
#27Earlier quoted context omitted.
The paper describes it pretty well in appendix C. A matrix of integrators is constructed with a bunch of opamps, RC time constants (using digital potentiometers, presumably) and a multichannel ADC/DAC interface to the PC. Essentially a dedicated differential-equation solver. So it's a combination of old-school analog computation and modern GPU-based code. Takes longer in practice due to the overhead of interfacing wi…
First author of the paper here. That's it indeed! One thing is that this is entirely CMOS-compatible. You could also do something similar with optics or other platforms, but we chose electronic circuits for this reason specifically.
Can you implement this on an FPGA?
Re: Thermodynamic Natural Gradient Descent
#28I don't get it, gradient descend computation is super frequent, state/input changes all the time, you'd have to reset heat landscape very frequently, what's the point? No way there is any potential speedup opportunity there, no?
If anything you could probably do something with electromagnetic fields, their interference, possibly in 3d.
Re: Thermodynamic Natural Gradient Descent
#29What's our current best guess of how animal neurons learn?
Re: Thermodynamic Natural Gradient Descent
#30Sounds great until > requires an analog thermodynamic computer Wait. What? Perhaps a trained physicist can comment on that. Thanks.
The paper describes it pretty well in appendix C. A matrix of integrators is constructed with a bunch of opamps, RC time constants (using digital potentiometers, presumably) and a multichannel ADC/DAC interface to the PC. Essentially a dedicated differential-equation solver. So it's a combination of old-school analog computation and modern GPU-based code. Takes longer in practice due to the overhead of interfacing wi…
Yes, digital wins over analog because of all the money that went into digital. I am wondering if one could create a digital analog computer by using pwm instead of analog signals.