Numerical Optimization: Understanding L-BFGS
1–10 of 15 posts
Re: Numerical Optimization: Understanding L-BFGS
#2Author here, happy to answer any questions.
Re: Numerical Optimization: Understanding L-BFGS
#3This is a nice introduction, but it looks there is a problem with the formatting: some of the math is not rendered properly (in the source, some expressions are wrapped in dollar-signs rather than script tags).
Re: Numerical Optimization: Understanding L-BFGS
#4Author here, happy to answer any questions.
Is it typical to interface to Newton's method via a function that computes the inverse Hessian? I've never seen that. Typically, people claim it is numerically unstable to explicitly invert the Hessian, and that it would be better for the interface to take the Hessian itself, and then call a subroutine to do a linear solve.
Re: Numerical Optimization: Understanding L-BFGS
#5Author here, happy to answer any questions.
I hate to be that guy, but "Raphson", not "Rhapson" (http://en.wikipedia.org/wiki/Joseph_Raphson). It's also worth noting that no one ever actually forms the inverse of H, even if H is dense. At worst you would compute some factorization of H and use that to solve for the update d.
Re: Numerical Optimization: Understanding L-BFGS
#6This is a nice introduction, but it looks there is a problem with the formatting: some of the math is not rendered properly (in the source, some expressions are wrapped in dollar-signs rather than script tags).
What browser are you in? It just using MathJax and appears to work in WebKit and Firefox.
Re: Numerical Optimization: Understanding L-BFGS
#7Author here, happy to answer any questions.
I hate to be that guy, but "Raphson", not "Rhapson" ( http://en.wikipedia.org/wiki/Joseph_Raphson ). It's also worth noting that no one ever actually forms the inverse of H, even if H is dense. At worst you would compute some factorization of H and use that to solve for the update d.
Typo is fixed, thanks!
I think that's explicitly mentioned in the Quasi-Newton section that you only need to implicitly multiply and not form the matrix.
Re: Numerical Optimization: Understanding L-BFGS
#8Earlier quoted context omitted.
I hate to be that guy, but "Raphson", not "Rhapson" ( http://en.wikipedia.org/wiki/Joseph_Raphson ). It's also worth noting that no one ever actually forms the inverse of H, even if H is dense. At worst you would compute some factorization of H and use that to solve for the update d.
Typo is fixed, thanks! I think that's explicitly mentioned in the Quasi-Newton section that you only need to implicitly multiply and not form the matrix.
I still see it misspelt throughout.
Re: Numerical Optimization: Understanding L-BFGS
#9Author here, happy to answer any questions.
Is it typical to interface to Newton's method via a function that computes the inverse Hessian? I've never seen that. Typically, people claim it is numerically unstable to explicitly invert the Hessian, and that it would be better for the interface to take the Hessian itself, and then call a subroutine to do a linear solve.
In practice you only need to do $H^{-1} g$; L-BFGS stores the {s_k} and {y_k} vectors which allow you to do the $H^{-1} g$ directly rather than needing to ever form the hessian or its inverse. There are techniques that aren't BFGS-based which approximate the hessian rather than the inverse and in that case you'd be better off solving.
Re: Numerical Optimization: Understanding L-BFGS
#10Earlier quoted context omitted.
Typo is fixed, thanks! I think that's explicitly mentioned in the Quasi-Newton section that you only need to implicitly multiply and not form the matrix.
I still see it misspelt throughout.
Want to try a refresh, I think it ought to be fixed now. Thanks