Only up to 64 variables? Why such small problems? And why are their results are averaged across all of the different test functions? I'd like to see the performance difference between Rosenbrock and Rastrigin, thank you very much. And they have a weird fixation on stopping rules, when in general your stopping rule is how many evaluations you can afford. Was this written by interns? There's no discussion of the retrog…
>Only up to 64 variables? Why such small problems?
This is a tool tuned for optimizing the hyperparameters of machine learning models. In practice, most models have, maybe, 10 hyperparameters in the normal sense (learning rate, nonlinearity, etc.). If you start talking about model structure as a hyperparameter, you can get vastly more than 64, but then these techniques aren't great anyway.
So why such small problems? Because the blackbox function you're optimizing takes hours to run. So as another user mentions, if it takes you 700 function evaluations, you're gonna be running for the better part of a year.
So the domain over which vizier works is one where you probably are never really using more than ~100 evaluations and often even then you would prefer to stop early if you meet certain conditions (because wasting significant compute doing unnecessary optimization is costly).
As a concrete example, a single iteration of a relatively small and non-SOTA algorithm takes 20 minutes and $40 to run. [0]
So stopping a day early saves you a day and $3000. Now scale that up 5 or 10x for larger models. (another way of putting this is that a day and $10K might be worth a .5% increase in accuracy, but isn't worth a .05% increase in accuracy. Stopping rules can encode that).