Beyond automatic differentiation
ai.googleblog.com
Beyond automatic differentiation
1–10 of 37 posts
Re: Beyond automatic differentiation
#2Interesting, but I have questions. The OP shows how SafeRate improves plain-vanilla Adam, but almost no one uses that anymore. The most commonly used optimizer nowadays is probably AdamW. Does SafeRate improve AdamW to the same extent, or at all? How does using SafeRate compare with Bayesian hyperparameter search? How does it compare with rule-of-thumb hyperparameter selection methods? Many people seem to get a lot of mileage with AdamW and sensible defaults (e.g., a "one cycle" schedule with a max lr of 3e-4, momentum peaking at 0.95, and some weight decay)?
Re: Beyond automatic differentiation
#3Interesting, but I have questions. The OP shows how SafeRate improves plain-vanilla Adam, but almost no one uses that anymore. The most commonly used optimizer nowadays is probably AdamW. Does SafeRate improve AdamW to the same extent, or at all? How does using SafeRate compare with Bayesian hyperparameter search? How does it compare with rule-of-thumb hyperparameter selection methods? Many people seem to get a lot o…
Lots of people still use Adam and SGD!
Re: Beyond automatic differentiation
#4Interesting, but I have questions. The OP shows how SafeRate improves plain-vanilla Adam, but almost no one uses that anymore. The most commonly used optimizer nowadays is probably AdamW. Does SafeRate improve AdamW to the same extent, or at all? How does using SafeRate compare with Bayesian hyperparameter search? How does it compare with rule-of-thumb hyperparameter selection methods? Many people seem to get a lot o…
Lots of people still use Adam and SGD!
Haven't seen anyone use it for large-scale (or largish-scale) learning in a long while...
Re: Beyond automatic differentiation
#5Can you define a hairy version of a function that misleads at every point? Ie
Take a well known function: sin(x). Call it G
Now make a function H that makes hairy version of functions. H takes a function f, a slope m, and a probability p.
H returns a version of the function where every value is the close to f(x), and p percent of the points have a derivative of m. The remaining points have the job of undoing the derivative of m and jumping back to f(x).
If you are dealing with real numbers then things can get very hairy without ever deviating from f(x). You could turn p up to 99.999999% and m to a crazy slope.
(I am not a mathematician, I don’t know if these musings are misguided)
My point is that if you had a hairy function, you wouldn’t be able to use gradient decent on it, because a derivative takes the slope at a point, and these slopes are misleading. However, you could use a numerical derivative, that plots the slope between a point and another point separated by a non zero distance D. The hairiness would not be detected.
Re: Beyond automatic differentiation
#6Can you define a hairy version of a function that misleads at every point? Ie Take a well known function: sin(x). Call it G Now make a function H that makes hairy version of functions. H takes a function f, a slope m, and a probability p. H returns a version of the function where every value is the close to f(x), and p percent of the points have a derivative of m. The remaining points have the job of undoing the deri…
Taking g(x) = f(x) + epsilon*cos(bx) adds to the derivatives a function that bounces between -b*epsilon and b*epsilon while g remains within an epsilon envelope of f. Take b large enough to fit your needs, e.g. 100/epsilon times an upper bound on |f'|.
Add a scaled version of the Weierstrass function instead if you don't want a derivative at all.
Re: Beyond automatic differentiation
#7My main question is whether this is still helpful in an SGD context?
I.e. if you have $f = \sum_i^n f_i$ and with every batch you make $f_S = \sum_{j\in S} f_j$ for that batch ... being able to take a step which definitely causes $f_S$ to decrease doesn't necessarily cause $f$ to decrease / the window for $f_S$ may be different than the window for $f$?
If it only really makes sense for cases where you can evaluate the whole function of interest at every step, which tend to be lower-dimensional anyways, then I think an interesting comparison would be to quasi-newton methods like LBGFS. I.e. if you're involving the taylor-series machinery, when should you use it to pick the direction to step (normal QN) vs when should you use it to pick only the step size (this work)?
Re: Beyond automatic differentiation
#8Can you define a hairy version of a function that misleads at every point? Ie Take a well known function: sin(x). Call it G Now make a function H that makes hairy version of functions. H takes a function f, a slope m, and a probability p. H returns a version of the function where every value is the close to f(x), and p percent of the points have a derivative of m. The remaining points have the job of undoing the deri…
I think for the application here (analogous to autograd on neural networks) you know each individual building block of the function (a well-defined neural network layer) so can be sure such things will not happen.
Re: Beyond automatic differentiation
#9I'm interested what fdej thinks about this one. arb has recently been merged with flint under his maintainership and is getting some generic rings wrapper and more interfaces with other languages. But maybe its focus is more like algebraic things like modular forms and less like tensor libraries for machine learning and GPGPU.
https://news.ycombinator.com/user?id=fdej
Re: Beyond automatic differentiation
#10My main question is whether this is still helpful in an SGD context? I.e. if you have $f = \sum_i^n f_i$ and with every batch you make $f_S = \sum_{j\in S} f_j$ for that batch ... being able to take a step which definitely causes $f_S$ to decrease doesn't necessarily cause $f$ to decrease / the window for $f_S$ may be different than the window for $f$? If it only really makes sense for cases where you can evaluate th…
The newton method has the nasty Wada property arise, which is often glossed over as being the same as scale invariant noise in class but which is better thought of as a topological feature rather than scale invariant noise.
It is indecomposable continua and not the imperfect knowledge of initial conditions like chaotic systems often invoke.
Boundary conditions becoming indeterminate is a problem with 3 or more attractors or exit basins.