Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A large scale non-linear optimization library

github.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: A large scale non-linear optimization library

#11

Post by dimatura »

margalabargala wrote: post #5

How does this compare to nlopt? https://nlopt.readthedocs.io/en/latest/

I believe Ceres was originally developed for nonlinear least squares optimization, particularly in the context of structure from motion (as bundle adjustment) and related problems in computer vision and robotics (esp. SLAM). As such it has some features that are useful in that context, such as automatic differentiation and covariance estimation. I see that it also has more general nonlinear optimization feature, but even in that feature it seems like it assumes there will be at least first order gradients available (and again, for this the auto differentiation is handy).

On the other hand it seems like NLOPT is oriented towards implementing various more general "black box" optimization methods, only some of which need/support gradient information, and there's no auto differentiation.

So if I was working on some kind of SFM/SLAM problem, I'd probably use Ceres, but if I had a less structured optimization problem - and especially if I didn't have gradients - I'd try NLOPT.

Re: A large scale non-linear optimization library

#12

Post by dimatura »

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

I believe that when the number of parameters isn't in the millions, as in deep networks, there's less of an advantage to using GPUs. But there is definitely research on using GPUs to solve nonlinear least squares optimization problems. Here's one I saw a few years ago: https://dl.acm.org/doi/10.1145/3132188

Re: A large scale non-linear optimization library

#14

Post by cerved »

zetazzed wrote: post #6
angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

For combinatorial optimization, there is now NVIDIA cuOpt, which is ridiculously fast: https://developer.nvidia.com/cuopt-logistics-optimization Ceres can use GPUs for some solver bits too, I believe.

I didn't know that, thank you. Is there any information about the internals of this? Like the kind of solver etc?

Re: A large scale non-linear optimization library

#15

Post by whatever1 »

angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

Sparse matrices is the answer. GPUs do not offer significant speed up in these cases.

Google for linear programming gpu acceleration to see for yourself

Re: A large scale non-linear optimization library

#16

Post by whatever1 »

zetazzed wrote: post #6
angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

For combinatorial optimization, there is now NVIDIA cuOpt, which is ridiculously fast: https://developer.nvidia.com/cuopt-logistics-optimization Ceres can use GPUs for some solver bits too, I believe.

Sorry but this is not optimization, this is a bunch of heuristics ran on a gpu.

Re: A large scale non-linear optimization library

#17

Post by angrycontrarian »

mgraczyk wrote: post #13
angrycontrarian wrote: post #10

Earlier quoted context omitted.

What about interior point methods?

They are iterative and path dependent, no?

Having implemented them myself on an ad hoc basis in pytorch, I don't see how they're much different than training a deep learning model.

Re: A large scale non-linear optimization library

#18

Post by cerved »

whatever1 wrote: post #16
zetazzed wrote: post #6

Earlier quoted context omitted.

For combinatorial optimization, there is now NVIDIA cuOpt, which is ridiculously fast: https://developer.nvidia.com/cuopt-logistics-optimization Ceres can use GPUs for some solver bits too, I believe.

Sorry but this is not optimization, this is a bunch of heuristics ran on a gpu.

What's your definition of optimization?

Re: A large scale non-linear optimization library

#19

Post by sampo »

whatever1 wrote: post #16
zetazzed wrote: post #6

Earlier quoted context omitted.

For combinatorial optimization, there is now NVIDIA cuOpt, which is ridiculously fast: https://developer.nvidia.com/cuopt-logistics-optimization Ceres can use GPUs for some solver bits too, I believe.

Sorry but this is not optimization, this is a bunch of heuristics ran on a gpu.

That's how you solve combinatorial optimization problems. With a bunch on heuristics.

Re: A large scale non-linear optimization library

#20

Post by cerved »

whatever1 wrote: post #15
angrycontrarian wrote: post #2

For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?

Sparse matrices is the answer. GPUs do not offer significant speed up in these cases. Google for linear programming gpu acceleration to see for yourself

I found this https://support.gurobi.com/hc/en-us/articles/360012237852-Do... which I guess is what you refer to?
Post reply on HN