A Practical Guide to Hyperparameter Optimization
blog.nanonets.com
A Practical Guide to Hyperparameter Optimization
1–10 of 35 posts
Re: A Practical Guide to Hyperparameter Optimization
#2It requires massive amount of computing power, otherwise theoretically you should be able to explore different optimizations automatically. Even then, validation is still hard and time consuming though.
Re: A Practical Guide to Hyperparameter Optimization
#3It requires massive amount of computing power, otherwise theoretically you should be able to explore different optimizations automatically. Even then, validation is still hard and time consuming though.
It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.
Re: A Practical Guide to Hyperparameter Optimization
#4It requires massive amount of computing power, otherwise theoretically you should be able to explore different optimizations automatically. Even then, validation is still hard and time consuming though.
It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.
Manually searching is time taking since you need to wait for the results from each experiment. This becomes impossible when the number of hyperparameters is more than 8-10 and you will probably end up only tuning a few of them that you think are relevant. You'd also need a lot of experience in tuning hyperparameters else your tuning is as good as random.
Given these disadvantages of manual tuning, "Bayesian Optimization" seems like the most promising technique, it needs a lot less "choose->train->evals" loops as it uses the information from previous runs to select the next set of hyperparameters (similar to what humans would do).
Re: A Practical Guide to Hyperparameter Optimization
#5Earlier quoted context omitted.
It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.
Manually searching is time taking since you need to wait for the results from each experiment. This becomes impossible when the number of hyperparameters is more than 8-10 and you will probably end up only tuning a few of them that you think are relevant. You'd also need a lot of experience in tuning hyperparameters else your tuning is as good as random. Given these disadvantages of manual tuning, "Bayesian Optimizat…
Does it work in parallel though?
Re: A Practical Guide to Hyperparameter Optimization
#6It requires massive amount of computing power, otherwise theoretically you should be able to explore different optimizations automatically. Even then, validation is still hard and time consuming though.
It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.
I am working on a little python framework to efficiently distribute hyperparameter search on a Spark cluster. We haven't released the first version yet but will do so in the next two weeks. https://github.com/logicalclocks/maggy
A limitation of existing hyperparameter search algorithms is that they are typically stage or generation-based. For example, if genetic algorithms are used for hyperparameter search, one has to wait for all models to finish in order to generate a new generation of potential parameters from the best performing individuals. However, some instances will have suboptimal parameters during a given iteration and will know quickly during the training that they can stop early. Hence, the early stopped machine can’t be provided with a new set of parameters early but is instead idle.
Compared to stage-based algorithms like genetic optimization algorithms, maggy (the framework) will support asynchronous algorithms, that are able to provide new candidate sets of parameters as soon as a worker finishes evaluating a combination and does not have to wait until all models in one stage finish. For this to be possible, we establish communication between the driver and executors in Spark. The driver will then collect performance metrics during training which enables us to stop badly performing models early during training and reassigning the executor task with a new, more promising set of parameters (new trial) right away, instead of waiting for a stage to finish.
Re: A Practical Guide to Hyperparameter Optimization
#7Is there a good reason not to regard this as a standard few-parameter no-gradient optimisation problem, and use something like Nelder-Mead on it?
Re: A Practical Guide to Hyperparameter Optimization
#8More essential background on Bayesian optimization in AutoML's HyperTune
https://cloud.google.com/blog/products/gcp/hyperparameter-tu...
Re: A Practical Guide to Hyperparameter Optimization
#9Earlier quoted context omitted.
Manually searching is time taking since you need to wait for the results from each experiment. This becomes impossible when the number of hyperparameters is more than 8-10 and you will probably end up only tuning a few of them that you think are relevant. You'd also need a lot of experience in tuning hyperparameters else your tuning is as good as random. Given these disadvantages of manual tuning, "Bayesian Optimizat…
Does it work in parallel though?
Sure, it does, it's not trivial though. Tedious to implement it yourself. You could use python libraries as "scikit-optimize" which has an implementation of parallel Bayesian optimization (based on Gaussian process), have a look at this: https://scikit-optimize.github.io/notebooks/bayesian-optimiz...
Re: A Practical Guide to Hyperparameter Optimization
#10More essential background on Bayesian optimization in AutoML's HyperTune https://cloud.google.com/blog/products/gcp/hyperparameter-tu...
Even Facebook uses Bayesian Optimization for tuning the parameters of some of its online systems (like its internal ranking system): https://research.fb.com/efficient-tuning-of-online-systems-u...