11–20 of 39 posts
I'm not an AI expert, but I'm decent with computers. Could anyone explain what is this about?
[1]: http://bair.berkeley.edu/blog/assets/saddle_eff/strictsaddle...
Many nonconvex problems are solved with more sophisticated methods, like L-BFGS. Are perturbations still a good thing?
> Our sharp rate depends on a key observation — although we don’t know the shape of the stuck region, we know it is very thin. Oh... really? :) (After 12 years I finally get an excuse to show a fun side project I coauthored during my PhD...) http://graemebell.net/pubs/taros05-bl-embedded-preprint.pdf Check out Figure 5 / Section 3.4 The rest of the paper is an introduction to why saddle points can be surprisingly pro…
> Our sharp rate depends on a key observation — although we don’t know the shape of the stuck region, we know it is very thin. Oh... really? :) (After 12 years I finally get an excuse to show a fun side project I coauthored during my PhD...) http://graemebell.net/pubs/taros05-bl-embedded-preprint.pdf Check out Figure 5 / Section 3.4 The rest of the paper is an introduction to why saddle points can be surprisingly pro…
These results seem important for nonconvex optimization in general, but for ML applications where we usually use stochastic/batch gradient descent, I wonder if the the stochasticity adds enough perturbation for this to not really be that useful.
Earlier quoted context omitted.
These results seem important for nonconvex optimization in general, but for ML applications where we usually use stochastic/batch gradient descent, I wonder if the the stochasticity adds enough perturbation for this to not really be that useful.
Well, in general, no, there is no perturbation method large (or good) enough to get out of saddle points, including via stochastic gradient descent. It might work in a particularly specific problem, but not in general.
I'm not an AI expert, but I'm decent with computers. Could anyone explain what is this about?
I'm not an AI expert either, but let me give this a try. I assume you are vaguely familiar with gradient descent. In gradient descent, we are basically trying to find the sweet spot where the value of a function is minimized. We do this by calculating the derivative of the function at a certain point and then use it to take small steps in the direction where we believe the function will have a lower value. Gradient d…
Earlier quoted context omitted.
Well, in general, no, there is no perturbation method large (or good) enough to get out of saddle points, including via stochastic gradient descent. It might work in a particularly specific problem, but not in general.
Fine, but I mostly care about ML applications, where I'm wondering if this is expected to help at all.
It's interesting, I once tried injecting perturbations for a short period selectively when the norm of the gradient was near zero, and it gave me consistent improvement.
How do you select the average size/standard deviation of your perturbations? Too small and you get no benefit, or very little speed up, too large and you lose the ability to sensibly optimise, and if you attempt to make an adaptive function then you'll find yourself with another postdoc.
