Are there any practical Pytorch examples? Say my network training time is 12 hours, I wonder how beneficial this would be for hyperparameter tuning over just simple grid/random search? Or would I instrument my network in a way to iterate over hyperparams faster than at every epoch/run?
Nevergrad: A Python library for performing derivative-free ML optimization
11–20 of 28 posts
Re: Nevergrad: A Python library for performing derivative-free ML optimization
#12Nevergrad?
I wonder if the maker never graduated.
;-)
Re: Nevergrad: A Python library for performing derivative-free ML optimization
#13Nevergrad? I wonder if the maker never graduated. ;-)
> Nevergrad offers an extensive collection of algorithms that do not require gradient computation
Re: Nevergrad: A Python library for performing derivative-free ML optimization
#14Earlier quoted context omitted.
It's black-box optimization. This means that we just have an objective function, without access to derivatives or whatever other information. This is not relevant for training weights in deep learning for image classification, or other things for which the gradient works well.
There was a recent paper from Uber, that GA works well for weights, so I wouldn't drop that area right away.
What’s GA here?
Re: Nevergrad: A Python library for performing derivative-free ML optimization
#15Re: Nevergrad: A Python library for performing derivative-free ML optimization
#16Are there any practical Pytorch examples? Say my network training time is 12 hours, I wonder how beneficial this would be for hyperparameter tuning over just simple grid/random search? Or would I instrument my network in a way to iterate over hyperparams faster than at every epoch/run?
We have not yet released examples of interfaces with Pytorch.
Maybe with moderate number of hyperparameters the benefit compared to random search will be moderate, whereas it will be very significant with high number of hyperparameters.
It also depends on how parallel you are. In all cases we have a wide range of algorithms with a common interface, so that you can compare.
We also use it for direct training of the weights of a network in reinforcement learning, not only hyperparameters.
Re: Nevergrad: A Python library for performing derivative-free ML optimization
#17Re: Nevergrad: A Python library for performing derivative-free ML optimization
#18Earlier quoted context omitted.
It's black-box optimization. This means that we just have an objective function, without access to derivatives or whatever other information. This is not relevant for training weights in deep learning for image classification, or other things for which the gradient works well.
There was a recent paper from Uber, that GA works well for weights, so I wouldn't drop that area right away.
Sure GA can be great for weights as well - but mainly when gradient is unreliable. I would not use Nevergrad for training the weights of a convolutional network for image classification for example; whereas I use Nevergrad for WorldModels.
Re: Nevergrad: A Python library for performing derivative-free ML optimization
#19Would this type of thing be suited for program synthesis or property based testing?
Re: Nevergrad: A Python library for performing derivative-free ML optimization
#20Are there any practical Pytorch examples? Say my network training time is 12 hours, I wonder how beneficial this would be for hyperparameter tuning over just simple grid/random search? Or would I instrument my network in a way to iterate over hyperparams faster than at every epoch/run?
We have not yet released examples of interfaces with Pytorch. Maybe with moderate number of hyperparameters the benefit compared to random search will be moderate, whereas it will be very significant with high number of hyperparameters. It also depends on how parallel you are. In all cases we have a wide range of algorithms with a common interface, so that you can compare. We also use it for direct training of the we…
Can you elaborate on the benefit for a high number of hyper parameters?