Example of thr bitter lesson. None of these nuanced matter 8 years later where everyone uses sgd or adamw.
An overview of gradient descent optimization algorithms (2016)
11–20 of 29 posts
Re: An overview of gradient descent optimization algorithms (2016)
#12Earlier quoted context omitted.
I disagree, it is old and most of those algorithms aren’t used anymore.
That’s how interviews go though, it’s not like I’ve ever had to use Bayes rule at work but for a few years everyone loved asking about it in screening rounds.
I'd still expect an MLE to know it though.
Re: An overview of gradient descent optimization algorithms (2016)
#13Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
I'm also curious about gradient-less algorithms For non deep learning applications, Nelder-Mead saved my butt a fees times
ChatGPT also advised me to use NM a couple of times, which was neat.
Re: An overview of gradient descent optimization algorithms (2016)
#14Earlier quoted context omitted.
I disagree, it is old and most of those algorithms aren’t used anymore.
That’s how interviews go though, it’s not like I’ve ever had to use Bayes rule at work but for a few years everyone loved asking about it in screening rounds.
In my experience a lot of people "know" maths, but fail to recognise the opportunities to use it. Some of my colleagues were pleased when I showed them that their ad hoc algorithm was equivalent to an application of Bayes' rule. It gave them insights into the meaning of constants that had formerly been chosen by trial and error.
Re: An overview of gradient descent optimization algorithms (2016)
#15Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
I'm also curious about gradient-less algorithms For non deep learning applications, Nelder-Mead saved my butt a fees times
Nelder–Mead has often not worked well for me in moderate to high dimensions. I'd recommend trying Powell's method if you want to quickly converge to a local optimum. If you're using scipy's wrappers, it's easy to swap between the two:
https://docs.scipy.org/doc/scipy/reference/optimize.html#loc...
For nastier optimization problems there are lots of other options, including evolutionary algorithms and Bayesian optimization:
Re: An overview of gradient descent optimization algorithms (2016)
#16Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.
I'm also curious about gradient-less algorithms For non deep learning applications, Nelder-Mead saved my butt a fees times
Look into zeroth-order optimizers and CMA-ES.
Re: An overview of gradient descent optimization algorithms (2016)
#17Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production. Messing with optimizers is one of the ways to enter hyperparameter hell: it’s like legacy code but on steroids because changing it only breaks your training code stochastically. Much better to stop worrying and love AdamW.
Luckily we have Shampoo, SOAP, Modula, Schedule-free variants, and many more these days being researched! I am very very excited by the heavyball library in particular
Re: An overview of gradient descent optimization algorithms (2016)
#18Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production. Messing with optimizers is one of the ways to enter hyperparameter hell: it’s like legacy code but on steroids because changing it only breaks your training code stochastically. Much better to stop worrying and love AdamW.
Something that stuck out to me in the updated blog [0] is that Demon Adam performed much better than even AdamW, with very interesting learning curves. I'm wondering now why it didn't become the standard. Anyone here have insights into this?
Re: An overview of gradient descent optimization algorithms (2016)
#19Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production. Messing with optimizers is one of the ways to enter hyperparameter hell: it’s like legacy code but on steroids because changing it only breaks your training code stochastically. Much better to stop worrying and love AdamW.
Something that stuck out to me in the updated blog [0] is that Demon Adam performed much better than even AdamW, with very interesting learning curves. I'm wondering now why it didn't become the standard. Anyone here have insights into this? [0] https://johnchenresearch.github.io/demon/
Demon Adam didn’t become standard largely for the same reason many “better” optimizers never see wide adoption: it’s a newer tweak, not clearly superior on every problem, is less familiar to most engineers, and isn’t always bundled in major frameworks. By contrast, AdamW is now the “safe default” that nearly everyone supports and knows how to tune, so teams stick with it unless they have a strong reason not to.
Edit: Demon involves decaying the momentum parameter over time, which introduces a new schedule or formula for how momentum should be reduced during training. That can feel like additional complexity or a potential hyperparameter rabbit hole. Teams trying to ship products quickly often avoid adding new hyperparameters unless the gains are decisive.
Re: An overview of gradient descent optimization algorithms (2016)
#20Earlier quoted context omitted.
I disagree, it is old and most of those algorithms aren’t used anymore.
That’s how interviews go though, it’s not like I’ve ever had to use Bayes rule at work but for a few years everyone loved asking about it in screening rounds.
Everyone’s experience is different but I’ve been in dozens of MLE interviews (some of which I passed!) and have never once been asked to explain the internals of an optimizer. The interviews were all post 2020, though.
Unless someone had a very good reason I would consider it weird to use anything other than AdamW. The compute you could save on a slightly better optimizer pale in comparison to the time you will spend debugging an opaque training bug.