NeurIPS 2020 Optimization Competition
bbochallenge.com
NeurIPS 2020 Optimization Competition
1–10 of 25 posts
Re: NeurIPS 2020 Optimization Competition
#2Interesting. There's been decades of research on Derivative-Free Optimization (DFO) and stochastic/evolutionary algorithms (most of which are derivative-free). They're used in practical applications, but have been hard to reliably perf benchmark because solution paths are so dependent on initial guess and random chance.
This one focuses on maximizing sample efficiency. That's an interesting (and important) metric to benchmark, especially for functions that are computationally expensive to evaluate, like full-on simulations. Sounds like the algorithm would need to be able to efficiently come up with an accurate surrogate model for the expensive function -- which is hard to do in the general case, but if something is known about the underlying function, some specialization is possible.
Re: NeurIPS 2020 Optimization Competition
#3Search is the problem that solves all other problems.
Re: NeurIPS 2020 Optimization Competition
#4NeurIPS, 2020: "We need more sample-efficient algorithms for finding better hyperparameters that specify how to train computationaly expensive deep learning models."
Rich Sutton, 2019: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." (https://news.ycombinator.com/item?id=23781400)
I wonder if in the end simply throwing more and more computation at the problem of finding good hyperparameters will end up working better as computation continues to get cheaper and cheaper.
Re: NeurIPS 2020 Optimization Competition
#5NeurIPS, 2020: "We need more sample-efficient algorithms for finding better hyperparameters that specify how to train computationaly expensive deep learning models." Rich Sutton, 2019: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." ( https://news.ycombinator.com/item?id=23781400 ) I wonder i…
It certainly does look that way for certain classes of problems, as witnessed by the evolution of GPT language models, where the model gets better through sheer use of compute resources.
For many combinatorial problems however, improvements in algorithms can often produce bigger strides than just throwing brute force compute at the problem. Take Mixed Integer Programs (MIPs) -- roughly the optimization-equivalent of SATs -- used for airline scheduling, optimal assignment problems and such. In slide 12 [1] (there are other sources that corroborate), the author notes that MIP solver performance between 1988-2017 had improved 2,527,768,000x.
17,120x was due to machine improvements (single core). 147,650x was due to algorithmic improvements. Multiple cores can also provide a performance boost up to a point, before saturating due to coordination costs. The author notes that "A typical MIP that would have taken 124 years to solve in 1988 will solve in 1 second now".
The biggest improvements in MIP algorithm performance have been due to improvements in solver heuristics (!), because the fastest computations are those that don't have to be performed at all -- i.e. that are eliminated via heuristics.
Re: NeurIPS 2020 Optimization Competition
#6NeurIPS, 2020: "We need more sample-efficient algorithms for finding better hyperparameters that specify how to train computationaly expensive deep learning models." Rich Sutton, 2019: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." ( https://news.ycombinator.com/item?id=23781400 ) I wonder i…
to do level-2 optimization you still have to run the level-1 problem repeatedly. So you'll only ever spend a small fraction of your resources on meta-optimization. The levels become exponentially more costly.
Re: NeurIPS 2020 Optimization Competition
#7NeurIPS, 2020: "We need more sample-efficient algorithms for finding better hyperparameters that specify how to train computationaly expensive deep learning models." Rich Sutton, 2019: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." ( https://news.ycombinator.com/item?id=23781400 ) I wonder i…
A very reasonable point and, certainly, the direction that parts of the computational community have embraced over the years. I will use integration as an example: classic computational methods were focused on trying to make strong assumptions about the integrand and significantly reduce the number of integrand evaluations (Gauss quadrature is the main thing that comes to mind). As computation became more accessible/parallelizable, and problems became less analytic, Monte Carlo methods have become more fundamental.
In some distributed computational settings, memory traffic is actually the main bottleneck and redundant computations are executed to reduce the need to send data (a similar situation to the one you aptly describe).
I think that, in the case of hyperparameter/meta-learning optimization (or search, depending on how you think about it) we are at a time right now where the complexity of models which can effectively be put into production is a function of our ability to, at least partially, analyze the space of possible modeling decisions. Will we escape that, and have models whose training cost is less significant than the cost of executing an "intelligent" hyperparameter search process? Maybe ... I am a GP person so I see potential in clever analysis of circumstances so that RKHS methods (for instance) can be leveraged and simplify the training process. But the current trajectory of the community has been to work on increasingly expensive models, which makes the ability to effectively use them with limited tuning/search cost still relevant.
Re: NeurIPS 2020 Optimization Competition
#8$5,000 for first place? That's not exactly enticing...
Re: NeurIPS 2020 Optimization Competition
#9if anyone wants to do this. i have a threadripper build with 2 2080ti. would be cool to do a group project.
write pm if you want. i am located in amsterdam, europe
Re: NeurIPS 2020 Optimization Competition
#10Search is the problem that solves all other problems.
back to square 1 :-)