21–30 of 35 posts
I'm using NNI[1] with decent success for hyperparameter optimization. It implements a number of different approaches, from a simple random search to a Tree Parzen Estimator (TPE) and specialized algorithms for automatically designing networks. It's very powerful and gives you a lot of freedom (it can minimize/maximize the output of fundamentally any python program). The main drawback is that you are on your own to fi…
Is there a good reason not to regard this as a standard few-parameter no-gradient optimisation problem, and use something like Nelder-Mead on it?
Bayesian parameter estimation typically trains an emulator to reproduce the objective function using a limited number of design point (order 10 per dimension). Once the emulator is trained, you could of course use a multi dimensional minimization function of your choice to find the best fit point. However, constructing and sampling the Bayesian posterior using MCMC methods has several advantages. Sometimes you can ha…
Big nearly-flat areas aren't really a new feature of hyperparameter problems... I guess the exact choice of algorithm would depend on how common they are, and maybe Nelder-Mead would be a poor choice. (And I'm not sure how easy it is to parallelise.)
I realize this is classic old-man-yells-at-cloud, but I don't understand why every online article these days, even the technical ones, need to have a giant "amusing" gif every two paragraphs. Do people not pay attention otherwise?
I realize this is classic old-man-yells-at-cloud, but I don't understand why every online article these days, even the technical ones, need to have a giant "amusing" gif every two paragraphs. Do people not pay attention otherwise?
Seems like the vast majority of the DL articles that make it the front of HN are just fluff. Nothing for DL practitioners, just 'hey look I can import tensorflow'.
Hi everyone. I'm the author. There's one more I thing I wanted to add: a good reason you should try using some sort of hyperparameter search, even you think it's a complete waste of time and compute, is for reproducibility. This probably applies more to open-source academic contributions, where you're trying to help your fellow practitioners recreate and use your models, as opposed to a corporate setting, where repro…
This explanation of momentum is somewhere between misleading and wrong. Momentum is about inertia and acceleration, i.e., the ability to quickly change speed.
I'm using NNI[1] with decent success for hyperparameter optimization. It implements a number of different approaches, from a simple random search to a Tree Parzen Estimator (TPE) and specialized algorithms for automatically designing networks. It's very powerful and gives you a lot of freedom (it can minimize/maximize the output of fundamentally any python program). The main drawback is that you are on your own to fi…
I'm used to hyperopt, do you know how they compare?
Is there a good reason not to regard this as a standard few-parameter no-gradient optimisation problem, and use something like Nelder-Mead on it?
I think many people (including the DFO community) already do that. People also consider the notion of multiple objectives important here I believe
[0]: https://devblogs.nvidia.com/sigopt-deep-learning-hyperparame...
More essential background on Bayesian optimization in AutoML's HyperTune https://cloud.google.com/blog/products/gcp/hyperparameter-tu...
[0]: https://sigopt.com/research/
[1]: https://sigopt.com/blog/covariance-kernels-for-avoiding-boun...
Hi everyone. I'm the author. There's one more I thing I wanted to add: a good reason you should try using some sort of hyperparameter search, even you think it's a complete waste of time and compute, is for reproducibility. This probably applies more to open-source academic contributions, where you're trying to help your fellow practitioners recreate and use your models, as opposed to a corporate setting, where repro…
> The heavier the ball, the quicker it falls. But if it’s too heavy, it can get stuck or overshoot the target. This explanation of momentum is somewhere between misleading and wrong. Momentum is about inertia and acceleration, i.e., the ability to quickly change speed.
Wasn't there a famous experiment about this someone once did?
I realize this is classic old-man-yells-at-cloud, but I don't understand why every online article these days, even the technical ones, need to have a giant "amusing" gif every two paragraphs. Do people not pay attention otherwise?
I noped out of there after seeing those and the terminator reference in the first paragraph. Maybe I'm not the target audience... Seems like the vast majority of the DL articles that make it the front of HN are just fluff. Nothing for DL practitioners, just 'hey look I can import tensorflow'.
Which is literally the ironic reference in the first image.