21–30 of 34 posts
For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?
We used it in a project that performed document recognition and analysis. It was useful for aligning the fields to be extracted from the source image as we formulated it as an optimization problem. It was straightforward to use and it contains some neat loss functions that reduce the effect of outliers (e.g. CauchyLoss).
If you prefer to reply privately, my contact details are in my HN profile.
For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?
How does this compare to nlopt? https://nlopt.readthedocs.io/en/latest/
I believe Ceres was originally developed for nonlinear least squares optimization, particularly in the context of structure from motion (as bundle adjustment) and related problems in computer vision and robotics (esp. SLAM). As such it has some features that are useful in that context, such as automatic differentiation and covariance estimation. I see that it also has more general nonlinear optimization feature, but…
- Estimate the pose of Street View cars, aircrafts, and satellites.
- Build 3D models for PhotoTours.
- Estimate satellite image sensor characteristics.
- Stitch panoramas on Android and iOS.
- Apply Lens Blur on Android.
- Solve bundle adjustment and SLAM problems in Project Tango.
Microsoft Research uses Ceres for nonlinear optimization of objectives involving subdivision surfaces under skinned control meshes.
For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?
For all of the success applying GPUs to optimization problems in ML, why don't any of the common optimization packages seem to support GPU acceleration?
More generally, there's a question of where the algorithms themselves benefit from GPUs or parallelism in general. For large scale nonlinear, continuous optimization problems using second-order, Newton like methods, the big costs are in the function evaluations, their derivatives, and the linear system preconditioners/solves. Generally speaking, how the function evaluations and derivatives are computed are on the user. For the preconditioning/linear system solves, there's value in parallelism. However, here, the GPUs have traditionally lagged. Basically, we need a factorization, be it sparse or dense, and it's only been recently where good library support has been extended for GPUs. For the longest time, the entire matrix factorization needed to fit onto the GPU and many of these matrices were large. That said, for optimizers that accept a user-defined preconditioner, the use of GPUs is already possible.