Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Practical Guide to Hyperparameter Optimization

blog.nanonets.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: A Practical Guide to Hyperparameter Optimization

#11

Post by wongarsu »

I'm using NNI[1] with decent success for hyperparameter optimization. It implements a number of different approaches, from a simple random search to a Tree Parzen Estimator (TPE) and specialized algorithms for automatically designing networks.

It's very powerful and gives you a lot of freedom (it can minimize/maximize the output of fundamentally any python program). The main drawback is that you are on your own to figure out which paramters go well together: For example using an assesor to stop underperforming attempts early is great for random search, but devastating for TPE. You have to figure that out on your own. You inevitably spend some time tuning your hyperparameter tuner. It's still a big win in terms of human effort, at the expense of doing a lot more computing.

1: https://github.com/Microsoft/nni

Re: A Practical Guide to Hyperparameter Optimization

#13

Post by bigred100 »

improbable22 wrote: post #7

Is there a good reason not to regard this as a standard few-parameter no-gradient optimisation problem, and use something like Nelder-Mead on it?

I think many people (including the DFO community) already do that. People also consider the notion of multiple objectives important here I believe

Re: A Practical Guide to Hyperparameter Optimization

#15

Post by improbable22 »

bigred100 wrote: post #13
improbable22 wrote: post #7

Is there a good reason not to regard this as a standard few-parameter no-gradient optimisation problem, and use something like Nelder-Mead on it?

I think many people (including the DFO community) already do that. People also consider the notion of multiple objectives important here I believe

Thanks. What's DFO? And what do you usefully do with multiple objectives, besides minimise some total?

Re: A Practical Guide to Hyperparameter Optimization

#16

Post by iyaja »

Hi everyone. I'm the author. There's one more I thing I wanted to add: a good reason you should try using some sort of hyperparameter search, even you think it's a complete waste of time and compute, is for reproducibility.

This probably applies more to open-source academic contributions, where you're trying to help your fellow practitioners recreate and use your models, as opposed to a corporate setting, where reproducibility would be the equivalent of getting fired.

Recently, I was trying to train a ResNet to beat the top Stanford DAWNBench entry (spoiler alert: I did, but by less than a second). Initially, I blindly tried manually tuning the learning rate, batch size, etc. without even reading the original model's guidelines.

After actually going through a blog post written by the David C Page (the guy with the top DAWNBench entry), I saw that he tried varying the hyperparameters himself and that the ones that were set by default in the code were what he found to be optimal.

That saved me a lot of time and let me focus on other things like what hardware to use.

I think the lesson here is that if more researchers perform and publish the results of some basic hyperparameter optimization, it would really save the world a whole lot of epochs.

Re: A Practical Guide to Hyperparameter Optimization

#17

Post by ovi256 »

jjn2009 wrote: post #3
stunt wrote: post #2

It requires massive amount of computing power, otherwise theoretically you should be able to explore different optimizations automatically. Even then, validation is still hard and time consuming though.

It sounds like an easy way to increase performance but really exploring the hyperparameter space is likely more efficiently done manually first and only automatically when you have figured out how to distribute the work.

>how to distribute the work

What do you mean by distribute the work ?

I've done hyperparameter searches manually, they're widely used in academic labs ("hyperparameter descent by grad student"), and I've also done a bit of hyperparameter automatic search, but I can't see what you meant.

Re: A Practical Guide to Hyperparameter Optimization

#18

Post by jcagalawan »

improbable22 wrote: post #15
bigred100 wrote: post #13

Earlier quoted context omitted.

I think many people (including the DFO community) already do that. People also consider the notion of multiple objectives important here I believe

Thanks. What's DFO? And what do you usefully do with multiple objectives, besides minimise some total?

DFO is derivative free optimization. With multiple objectives you try to find different solutions given different weightings to the objectives for the Pareto front and pick one depending on the domain.

Re: A Practical Guide to Hyperparameter Optimization

#19

Post by besulzbach »

iyaja wrote: post #16

Hi everyone. I'm the author. There's one more I thing I wanted to add: a good reason you should try using some sort of hyperparameter search, even you think it's a complete waste of time and compute, is for reproducibility. This probably applies more to open-source academic contributions, where you're trying to help your fellow practitioners recreate and use your models, as opposed to a corporate setting, where repro…

I enjoyed the article, and I know that writing these takes a nontrivial amount of time. So I think it would be wise of you to run these through a spell checker before publishing, as this is a less than a minute investment which pays off every time someone reads it.

Re: A Practical Guide to Hyperparameter Optimization

#20

Post by tictacttoe »

improbable22 wrote: post #7

Is there a good reason not to regard this as a standard few-parameter no-gradient optimisation problem, and use something like Nelder-Mead on it?

Bayesian parameter estimation typically trains an emulator to reproduce the objective function using a limited number of design point (order 10 per dimension). Once the emulator is trained, you could of course use a multi dimensional minimization function of your choice to find the best fit point.

However, constructing and sampling the Bayesian posterior using MCMC methods has several advantages. Sometimes you can have a local minimum which is essentially flat, so the optimal hyperparameter is unstable. You'll see this in the posterior distribution. Or you could have two parameters which are correlated so it's their sum that's constrained not their individual values. All this information provides important context when understanding your model's uncertainty.

Post reply on HN