Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An overview of gradient descent optimization algorithms (2016)

ruder.io

Post reply on HN
Print view View original

21–29 of 29 posts

Re: An overview of gradient descent optimization algorithms (2016)

#21

Post by analog31 »

VHRanger wrote: post #10
sega_sai wrote: post #3

Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.

I'm also curious about gradient-less algorithms For non deep learning applications, Nelder-Mead saved my butt a fees times

It's with the utmost humility that I confess to falling back on "just use Nelder-Mead" in *scipy.optimize* when something is ill behaved. I consider it to be a sign that I'm doing something wrong, but I certainly respect its use.

Re: An overview of gradient descent optimization algorithms (2016)

#22

Post by kernc »

VHRanger wrote: post #10
sega_sai wrote: post #3

Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.

I'm also curious about gradient-less algorithms For non deep learning applications, Nelder-Mead saved my butt a fees times

SAMBO does a good job of finding the global optimum in a black-box manner even compared to Nelder-Mead, according to its own benchmark ...

https://sambo-optimization.github.io

Re: An overview of gradient descent optimization algorithms (2016)

#23

Post by juliangoldsmith »

mike-the-mikado wrote: post #7
sega_sai wrote: post #3

Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.

I think the big difference is dimensionality. If the dimensionality is low, then taking account of the 2nd derivatives becomes practical and worthwhile.

What is it that makes higher order derivatives less useful at high dimensionality? Is it related to the Curse of Dimensionality, or maybe something like exploding gradients at higher orders?

Re: An overview of gradient descent optimization algorithms (2016)

#24

Post by janalsncm »

esafak wrote: post #12
sk11001 wrote: post #8

Earlier quoted context omitted.

That’s how interviews go though, it’s not like I’ve ever had to use Bayes rule at work but for a few years everyone loved asking about it in screening rounds.

I'd still expect an MLE to know it though.

Why would you? Implementing optimizers isn’t something that MLEs do. Even the Deepseek team just uses AdamW.

An MLE should be able to look up and understand the differences between optimizers but memorizing that information is extremely low priority compared with other information they might be asked.

Re: An overview of gradient descent optimization algorithms (2016)

#25

Post by yobbo »

janalsncm wrote: post #20
sk11001 wrote: post #8

Earlier quoted context omitted.

That’s how interviews go though, it’s not like I’ve ever had to use Bayes rule at work but for a few years everyone loved asking about it in screening rounds.

Everyone’s experience is different but I’ve been in dozens of MLE interviews (some of which I passed!) and have never once been asked to explain the internals of an optimizer. The interviews were all post 2020, though. Unless someone had a very good reason I would consider it weird to use anything other than AdamW. The compute you could save on a slightly better optimizer pale in comparison to the time you will spend…

For example, if it is meaningful to use large batch sizes, the gradient variance will be lower and adam could be equivalent to just momentum.

As a model is trained, the gradient variance typically falls.

Those optimizers all work to reduce the variance of the updates in various ways.

Re: An overview of gradient descent optimization algorithms (2016)

#26

Post by mike-the-mikado »

juliangoldsmith wrote: post #23
mike-the-mikado wrote: post #7

Earlier quoted context omitted.

I think the big difference is dimensionality. If the dimensionality is low, then taking account of the 2nd derivatives becomes practical and worthwhile.

What is it that makes higher order derivatives less useful at high dimensionality? Is it related to the Curse of Dimensionality, or maybe something like exploding gradients at higher orders?

In n dimensions, the first derivative is an n-element vector. The second derivative is an n x n (symmetric) matrix. As n grows, the computation required to estimate the matrix increases (as at least n^2) and computation needed to use it increases (possibly faster).

In practice, clever optimisation algorithms that use the 2nd derivative won't actually form this matrix.

Re: An overview of gradient descent optimization algorithms (2016)

#27

Post by [deleted] »

kernc wrote: post #22
VHRanger wrote: post #10

Earlier quoted context omitted.

I'm also curious about gradient-less algorithms For non deep learning applications, Nelder-Mead saved my butt a fees times

SAMBO does a good job of finding the global optimum in a black-box manner even compared to Nelder-Mead, according to its own benchmark ... https://sambo-optimization.github.io

[deleted]

Re: An overview of gradient descent optimization algorithms (2016)

#28

Post by 3abiton »

pizza wrote: post #17
janalsncm wrote: post #4

Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production. Messing with optimizers is one of the ways to enter hyperparameter hell: it’s like legacy code but on steroids because changing it only breaks your training code stochastically. Much better to stop worrying and love AdamW.

Luckily we have Shampoo, SOAP, Modula, Schedule-free variants, and many more these days being researched! I am very very excited by the heavyball library in particular

Been out of the loop for while, anything exciting?

Re: An overview of gradient descent optimization algorithms (2016)

#29

Post by pizza »

3abiton wrote: post #28
pizza wrote: post #17

Earlier quoted context omitted.

Luckily we have Shampoo, SOAP, Modula, Schedule-free variants, and many more these days being researched! I am very very excited by the heavyball library in particular

Been out of the loop for while, anything exciting?

Read the Modular Norms in Deep Learning paper, and follow the author of the heavyball library on twitter with notifications enabled
Post reply on HN