An overview of gradient descent optimization algorithms (2016)
1–10 of 29 posts
Re: An overview of gradient descent optimization algorithms (2016)
#2It's a great summary for ML interview prep.
Re: An overview of gradient descent optimization algorithms (2016)
#3Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
Re: An overview of gradient descent optimization algorithms (2016)
#4Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production.
Messing with optimizers is one of the ways to enter hyperparameter hell: it’s like legacy code but on steroids because changing it only breaks your training code stochastically. Much better to stop worrying and love AdamW.
Re: An overview of gradient descent optimization algorithms (2016)
#5It's a great summary for ML interview prep.
I disagree, it is old and most of those algorithms aren’t used anymore.
Re: An overview of gradient descent optimization algorithms (2016)
#6[deleted]
Re: An overview of gradient descent optimization algorithms (2016)
#7Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
I think the big difference is dimensionality. If the dimensionality is low, then taking account of the 2nd derivatives becomes practical and worthwhile.
Re: An overview of gradient descent optimization algorithms (2016)
#8Re: An overview of gradient descent optimization algorithms (2016)
#9Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production. Messing with optimizers is one of the ways to enter hyperparameter hell: it’s like legacy code but on steroids because changing it only breaks your training code stochastically. Much better to stop worrying and love AdamW.
The mention of AdamW is brief, but in his defense he includes a link that gives a gloss of it: "An updated overview of recent gradient descent algorithms" [https://johnchenresearch.github.io/demon/].
Re: An overview of gradient descent optimization algorithms (2016)
#10Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
I'm also curious about gradient-less algorithms
For non deep learning applications, Nelder-Mead saved my butt a fees times