Gradient Descent Optimisation Algorithms
towardsdatascience.com
Gradient Descent Optimisation Algorithms
1–10 of 17 posts
Re: Gradient Descent Optimisation Algorithms
#2At the end of this post you will get a cheat sheet of the 10 common gradient descent optimisation algorithms.
Using more readable notations, I will walk you through how the vanilla stochastic gradient descent slowly evolved into the popular Adam optimiser and others. I also came out with an ‘evolutionary map’ of the optimisers to visualise this.
The motivation for writing this post is that there is a lack of simple-to-read equations for parameter update and a compiled list of these optimisers.
Hopefully this benefits the community.
Re: Gradient Descent Optimisation Algorithms
#3This is very helpful! If I may make a shameless self-plug, this would be even better as something that is dynamic and can be interactively played with. A few years ago I made this iPython notebook for similar didactic purposes: https://github.com/turingbirds/gradient_descent/blob/master/...
Re: Gradient Descent Optimisation Algorithms
#4Great summarization! Saved a lot of googling for me. Thanks for this.
Re: Gradient Descent Optimisation Algorithms
#5For those unfamiliar with the concept, courtesy Wikipedia
https://en.m.wikipedia.org/wiki/Gradient_descent
The basic intuition behind gradient descent can be illustrated by a hypothetical scenario. A person is stuck in the mountains and is trying to get down (i.e. trying to find the minima). There is heavy fog such that visibility is extremely low. Therefore, the path down the mountain is not visible, so he must use local information to find the minima. He can use the method of gradient descent, which involves looking at the steepness of the hill at his current position, then proceeding in the direction with the steepest descent (i.e. downhill). If he was trying to find the top of the mountain (i.e. the maxima), then he would proceed in the direction steepest ascent (i.e. uphill). Using this method, he would eventually find his way down the mountain. However, assume also that the steepness of the hill is not immediately obvious with simple observation, but rather it requires a sophisticated instrument to measure, which the person happens to have at the moment. It takes quite some time to measure the steepness of the hill with the instrument, thus he should minimize his use of the instrument if he wanted to get down the mountain before sunset. The difficulty then is choosing the frequency at which he should measure the steepness of the hill so not to go off track.
Re: Gradient Descent Optimisation Algorithms
#6Gradient descent is the abstraction of so many real world problems it's not even funny. From protein folding to machine intelligence, gradient descents everywhere...
Re: Gradient Descent Optimisation Algorithms
#7This is very helpful! If I may make a shameless self-plug, this would be even better as something that is dynamic and can be interactively played with. A few years ago I made this iPython notebook for similar didactic purposes: https://github.com/turingbirds/gradient_descent/blob/master/...
This is great! I'd love to do something similar in JavaScript.
Re: Gradient Descent Optimisation Algorithms
#8If I may, I had also built a simple demo of linear regression using gradient descent before writing this post. https://raiboso.me/backpropagation-demo/
This demo allows you to choose between four optimisers, and lets you track the values of your variables during training.
Compare your runs with different optimisers using the graph at the bottom of the page.
Re: Gradient Descent Optimisation Algorithms
#9And for anyone who wants to know why unmodified gradient descent may be considered a piece of shit in certain circumstances
http://wikipedia.org/wiki/Rosenbrock_function
Gradient descent with a good line search (Wolfe conditions) applies to the multidimensional case should converge to min, but it might take you thousands of iterations. Newton’s method or something might take But machine learning practitioners will know why gradient algorithms are often preferred despite this
Re: Gradient Descent Optimisation Algorithms
#10And for anyone who wants to know why unmodified gradient descent may be considered a piece of shit in certain circumstances http://wikipedia.org/wiki/Rosenbrock_function Gradient descent with a good line search (Wolfe conditions) applies to the multidimensional case should converge to min, but it might take you thousands of iterations. Newton’s method or something might take But machine learning practitioners will kn…
I'm not in ML, but one thing I remember from school (a million years ago, before ML was big) is that Newton's method needs a matrix inversion every iteration, which is expensive when you have a lot of decision variables. Not sure why other old school algorithms like conjugate gradient aren't used though.
I guess for rectified linear activation functions the second derivative isn't useful. Maybe that's it.