A large scale non-linear optimization library

#1

Post by rch » Thu, Oct 20, 2022, 2:47 AM UTC

A large scale non-linear optimization library
github.com

Re: A large scale non-linear optimization library

#2

Post by angrycontrarian » Thu, Oct 20, 2022, 5:11 AM UTC

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

Re: A large scale non-linear optimization library

#3

Post by mrfox321 » Thu, Oct 20, 2022, 5:17 AM UTC

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

Less users. Smaller matrices.

Re: A large scale non-linear optimization library

#4

Post by angrycontrarian » Thu, Oct 20, 2022, 5:23 AM UTC

mrfox321 wrote: post #3

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

Less users. Smaller matrices.

There are a lot of large-scale optimization problems in industry that are still compute bound. Currently available solvers are either single threaded on the CPU, or offer "parallelism" by running copies of the same problem on multiple threads, but with different initial conditions in hopes that one happens to converge faster.

Re: A large scale non-linear optimization library

#5

Post by margalabargala » Thu, Oct 20, 2022, 5:24 AM UTC

How does this compare to nlopt?

https://nlopt.readthedocs.io/en/latest/

Re: A large scale non-linear optimization library

#6

Post by zetazzed » Thu, Oct 20, 2022, 5:30 AM UTC

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

For combinatorial optimization, there is now NVIDIA cuOpt, which is ridiculously fast: https://developer.nvidia.com/cuopt-logistics-optimization

Ceres can use GPUs for some solver bits too, I believe.

Re: A large scale non-linear optimization library

#7

Post by mgaunard » Thu, Oct 20, 2022, 5:34 AM UTC

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

Optimization is an iterative path-dependent problem, it's not particularly GPU-friendly.

Re: A large scale non-linear optimization library

#8

Post by [deleted] » Thu, Oct 20, 2022, 5:37 AM UTC

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

[deleted]

Re: A large scale non-linear optimization library

#9

Post by hackerlight » Thu, Oct 20, 2022, 5:39 AM UTC

angrycontrarian wrote: post #4

mrfox321 wrote: post #3

Earlier quoted context omitted.

Less users. Smaller matrices.

There are a lot of large-scale optimization problems in industry that are still compute bound. Currently available solvers are either single threaded on the CPU, or offer "parallelism" by running copies of the same problem on multiple threads, but with different initial conditions in hopes that one happens to converge faster.

Most large-scale optimization outside of neural nets is bottlenecked by function evaluation

Re: A large scale non-linear optimization library

#10

Post by angrycontrarian » Thu, Oct 20, 2022, 5:40 AM UTC

mgaunard wrote: post #7

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

Optimization is an iterative path-dependent problem, it's not particularly GPU-friendly.

What about interior point methods?