Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beyond automatic differentiation

ai.googleblog.com

Post reply on HN
Print view View original

11–20 of 37 posts

Re: Beyond automatic differentiation

#11

Post by modeless »

> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting).

I'm not a huge fan of analyses like this that make one huge assumption that never holds in practice. Nobody uses full-batch gradient descent. It's not possible in practice or even desirable in theory. The stochasticity of SGD is important for performance! There is no reason to expect results with unrealistic assumptions like these to generalize to the cases people actually care about.

> at the cost of a single additional forward pass that increases the wall time for each step by a small factor (about 2x in the example above).

So it doubles the training time and FLOPS. Then the graph X axis should be wall time or FLOPS when comparing the methods, instead of steps.

Re: Beyond automatic differentiation

#12

Post by GaggiX »

cs702 wrote: post #2

Interesting, but I have questions. The OP shows how SafeRate improves plain-vanilla Adam, but almost no one uses that anymore. The most commonly used optimizer nowadays is probably AdamW. Does SafeRate improve AdamW to the same extent, or at all? How does using SafeRate compare with Bayesian hyperparameter search? How does it compare with rule-of-thumb hyperparameter selection methods? Many people seem to get a lot o…

It improves Adam in the full-batch setting, which is not usually done or desired since minibatch should regularize and improve the loss.

Re: Beyond automatic differentiation

#13

Post by [deleted] »

currymj wrote: post #8
sagebird wrote: post #5

Can you define a hairy version of a function that misleads at every point? Ie Take a well known function: sin(x). Call it G Now make a function H that makes hairy version of functions. H takes a function f, a slope m, and a probability p. H returns a version of the function where every value is the close to f(x), and p percent of the points have a derivative of m. The remaining points have the job of undoing the deri…

I think for the application here (analogous to autograd on neural networks) you know each individual building block of the function (a well-defined neural network layer) so can be sure such things will not happen.

[deleted]

Re: Beyond automatic differentiation

#14

Post by ogogmad »

Haven't looked in any depth: How does this cope with functions like f(x) = x - x? Interval arithmetic sometimes produces very conservative bounds.

If done well, rigorous interval bounds seem like they could be really useful.

Re: Beyond automatic differentiation

#15

Post by GistNoesis »

modeless wrote: post #11

> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting). I'm not a huge fan of analyses like this that make one huge assumption that never holds in practice. Nobody uses full-batch gradient descent. It's not possible in practice or even desirable in theory. The stochasticity…

In the paper https://arxiv.org/pdf/2212.11429.pdf In point 5.5 (page 57) Future work :

"

Promising areas of future work include: • Mini-batch optimization. Both the theory and experiments in this chapter have been limited to full- batch optimization. However, the MM paradigm can be extended to mini-batch optimization (e.g., [54]). A universal mini-batch MM optimization algorithm may outperform Adam and AdaGrad on certain large-scale machine learning problems.

"

The current limits of this algorithm (point 2 page 56) is that the reverse mode bounding is not implemented yet (and related work didn't obtain tight bounds) so you can't yet apply it directly for function with huge number of input parameters, so for the time being the main usage is limited to few hyper-parameter optimizations.

Re: Beyond automatic differentiation

#16

Post by fn-mote »

modeless wrote: post #11

> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting). I'm not a huge fan of analyses like this that make one huge assumption that never holds in practice. Nobody uses full-batch gradient descent. It's not possible in practice or even desirable in theory. The stochasticity…

> So it doubles the training time and FLOPS. Then the graph X axis should be wall time or FLOPS when comparing the methods, instead of steps.

Thanks for pointing this out.

Inflating your results by tricky measurement methods has been around forever, but it can be hard to spot if you're not close enough to the field.

Also, unfortunately, the blog post does not contain anything about the key idea used. (Well, I couldn't find it.)

Trust regions have been around forever; that's not the new idea.

It would be nice if somewhere there was a piece about "this is what we do better". Decoding a scientific article to figure out how much fluff it is seems like it has an uncertain return.

Re: Beyond automatic differentiation

#17

Post by fdej »

ftxbro wrote: post #9

I'm interested what fdej thinks about this one. arb has recently been merged with flint under his maintainership and is getting some generic rings wrapper and more interfaces with other languages. But maybe its focus is more like algebraic things like modular forms and less like tensor libraries for machine learning and GPGPU. https://news.ycombinator.com/user?id=fdej https://news.ycombinator.com/item?id=26054520 htt…

This looks quite similar like the "Lagrange models" defined by Joris van der Hoeven in https://hal.science/hal-01188378/document, a version of Taylor models where the constant error term is replaced by an interval multiple of x^n.

Representing functions locally is a very interesting problem: there are lots of possible approaches using intervals, Taylor series, Chebyshev series, etc. Big open design space if you ask me.

Re: Beyond automatic differentiation

#18

Post by kelseyfrog »

That's great. Interval arithmetic is an underappreciated area of study, not because it's particularly difficult - anyone can grasp it in short order, but it's rarely the first, second, or third tool used for the job when confronted with a problem. Perhaps it's just not prestigious enough compared to more sophisticated methods.

Re: Beyond automatic differentiation

#19

Post by ogogmad »

ftxbro wrote: post #9

I'm interested what fdej thinks about this one. arb has recently been merged with flint under his maintainership and is getting some generic rings wrapper and more interfaces with other languages. But maybe its focus is more like algebraic things like modular forms and less like tensor libraries for machine learning and GPGPU. https://news.ycombinator.com/user?id=fdej https://news.ycombinator.com/item?id=26054520 htt…

I've just looked at the documentation for eigenvalues and eigenvectors in arblib. I was thinking that:

- Eigenvectors are not computable in regions surrounding eigenvalue clashes. Therefore you should never compute them.

- Let M be a symmetric matrix being eigen-decomposed. The decomposition should be A = P D P^(-1) where only D is interval-valued, and P and P^(-1) are NOT UNDER ANY CIRCUMSTANCE INTERVAL-VALUED, and A should include M.

Re: Beyond automatic differentiation

#20

Post by ogogmad »

fdej wrote: post #17
ftxbro wrote: post #9

I'm interested what fdej thinks about this one. arb has recently been merged with flint under his maintainership and is getting some generic rings wrapper and more interfaces with other languages. But maybe its focus is more like algebraic things like modular forms and less like tensor libraries for machine learning and GPGPU. https://news.ycombinator.com/user?id=fdej https://news.ycombinator.com/item?id=26054520 htt…

This looks quite similar like the "Lagrange models" defined by Joris van der Hoeven in https://hal.science/hal-01188378/document , a version of Taylor models where the constant error term is replaced by an interval multiple of x^n. Representing functions locally is a very interesting problem: there are lots of possible approaches using intervals, Taylor series, Chebyshev series, etc. Big open design space if you ask…

See my comment above about your linear algebra implementation.
Post reply on HN