A Practical Guide to Hyperparameter Optimization • Hacker News Board

Re: A Practical Guide to Hyperparameter Optimization

#31

Post by mccourt » Thu, Apr 18, 2019, 5:21 PM UTC

I always appreciate articles emphasizing the importance of hyperparameter optimization; thank you for writing this. The discussion on learning rate is nice additional point to mention, though I find it a bit misleading -- earlier in the discussion you are mentioning a number of hyperparameters but then learning rate is studied in a vacuum. If other hyperparameters were varied along with the learning rate, I assume those graphics would look much more complicated.

Additionally, practical circumstances for hyperparameter tuning using Bayesian optimization often include complications: dealing with discrete hyperparameters, large parameter spaces being unreasonably costly or poorly modeled, accounting for uncertainty in your metric, balancing competing metrics, black-box constraints. Obviously, one cannot mention everything in a blog post, I just wanted to bring up that outstanding researchers in Bayesian optimization are pushing forward on all of these topics.

Regardless, thank you for continuing to hammer home the value of hyperparameter optimization. If I may, a couple links, for anyone trying to learn more:

My favorite BO intro - https://arxiv.org/abs/1807.02811 AutoML from the Freiburg crew - http://papers.nips.cc/paper/5872-efficient-and-robust-automa... Some discussion on parallelism/high dimensions - https://bayesopt.github.io/papers/2017/3.pdf Strategies for warm starting - https://ml.informatik.uni-freiburg.de/papers/18-AUTOML-RGPE....

Re: A Practical Guide to Hyperparameter Optimization

#32

Post by jjn2009 » Thu, Apr 18, 2019, 5:48 PM UTC

moritzmeister wrote: post #6

jjn2009 wrote: post #3

Earlier quoted context omitted.

It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.

I am working on a little python framework to efficiently distribute hyperparameter search on a Spark cluster. We haven't released the first version yet but will do so in the next two weeks. https://github.com/logicalclocks/maggy A limitation of existing hyperparameter search algorithms is that they are typically stage or generation-based. For example, if genetic algorithms are used for hyperparameter search, one has…

Nice. I've been using hyperopt with random search, I'll definitely check out your work.

Re: A Practical Guide to Hyperparameter Optimization

#33

Post by jjn2009 » Thu, Apr 18, 2019, 5:52 PM UTC

ovi256 wrote: post #17

jjn2009 wrote: post #3

Earlier quoted context omitted.

It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.

>how to distribute the work What do you mean by distribute the work ? I've done hyperparameter searches manually, they're widely used in academic labs ("hyperparameter descent by grad student"), and I've also done a bit of hyperparameter automatic search, but I can't see what you meant.

"hyperparameter descent by grad student" is a lot more efficient at first, much of the time the loss function has caveats which make it easy to fall into parts of the search space which don't actually accomplish the task (for example when empty frame = true reduces the loss) something a grad student would easily figure out. Until you get to the point where you are fairly certain the search is within the right space its hard to ensure that throwing a lot of compute at that search will yield anything useful.

Re: A Practical Guide to Hyperparameter Optimization

#34

Post by jjn2009 » Thu, Apr 18, 2019, 6:05 PM UTC

manneshiva wrote: post #4

jjn2009 wrote: post #3

Earlier quoted context omitted.

It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.

Manually searching is time taking since you need to wait for the results from each experiment. This becomes impossible when the number of hyperparameters is more than 8-10 and you will probably end up only tuning a few of them that you think are relevant. You'd also need a lot of experience in tuning hyperparameters else your tuning is as good as random. Given these disadvantages of manual tuning, "Bayesian Optimizat…

It depends on how well the problem is understood. If the problem is your standard MNIST dataset then sure it could very well be a waste of time to sit around and serialize your manual hyper param search. For any new datasets which may or may not be cleaned theres much to be learned from iterating on a very small subset of the data, at that small scale it's much easier to get a handle on the major failings, such as encoding the wrong things or weight explosion.

Re: A Practical Guide to Hyperparameter Optimization

#35

Post by wongarsu » Thu, Apr 18, 2019, 10:51 PM UTC

polynomial wrote: post #29

d__k wrote: post #25

Earlier quoted context omitted.

> The heavier the ball, the quicker it falls. But if it’s too heavy, it can get stuck or overshoot the target. This explanation of momentum is somewhere between misleading and wrong. Momentum is about inertia and acceleration, i.e., the ability to quickly change speed.

> The heavier the ball, the quicker it falls Wasn't there a famous experiment about this someone once did?

A misleading experiment: the heavier the feather the quicker it falls is obviously true; steel feathers are useless. The same is true for balls (except for the exceptional situation of a perfect vacuum), it's just that drag and air currents don't influence balls all that much at low speeds.