Optimizing ML training with metagradient descent
1–10 of 14 posts
Re: Optimizing ML training with metagradient descent
#2In the ML problem I'm working on now, there are about a dozen simple hyperparameters, and each training run takes hours or even days. I don't think there's any good way to search the space of hyperparameters without a deep understanding of the problem domain, and even then I'm often surprised when a minor config tweak yields better results (or fails to). Many of these hyperparameters affect performance directly and are very sensitive to hardware limits, so a bad value leads to an out-of-memory error in one direction or a runtime measured in years in the other. It's a real-world halting problem on steroids.
This is not to even mention more complex design decisions, like the architecture of the model, which can't be captured in a simple hyperparameter.
Re: Optimizing ML training with metagradient descent
#3In the ML problem I'm working on now, there are about a dozen simple hyperparameters, and each training run takes hours or even days. I don't think there's any good way to search the space of hyperparameters without a deep understanding of the problem domain, and even then I'm often surprised when a minor config tweak yields better results (or fails to). Many of these hyperparameters affect performance directly and a…
You might find this helpful for prioritizing which knobs to turn first
https://github.com/google-research/tuning_playbook
Re: Optimizing ML training with metagradient descent
#4How does it compare to previous work on learning to learn? I don't see it referenced https://arxiv.org/abs/1606.04474
Re: Optimizing ML training with metagradient descent
#5In the ML problem I'm working on now, there are about a dozen simple hyperparameters, and each training run takes hours or even days. I don't think there's any good way to search the space of hyperparameters without a deep understanding of the problem domain, and even then I'm often surprised when a minor config tweak yields better results (or fails to). Many of these hyperparameters affect performance directly and a…
You might find this helpful for prioritizing which knobs to turn first https://github.com/google-research/tuning_playbook
Starting to get a bit out of date. Pity they stopped updating it.
Re: Optimizing ML training with metagradient descent
#6In the ML problem I'm working on now, there are about a dozen simple hyperparameters, and each training run takes hours or even days. I don't think there's any good way to search the space of hyperparameters without a deep understanding of the problem domain, and even then I'm often surprised when a minor config tweak yields better results (or fails to). Many of these hyperparameters affect performance directly and a…
Are you already employing Bayesian optimization techniques? These are commonly used to explore spaces where evaluation is expensive.
Re: Optimizing ML training with metagradient descent
#7In the ML problem I'm working on now, there are about a dozen simple hyperparameters, and each training run takes hours or even days. I don't think there's any good way to search the space of hyperparameters without a deep understanding of the problem domain, and even then I'm often surprised when a minor config tweak yields better results (or fails to). Many of these hyperparameters affect performance directly and a…
Are you already employing Bayesian optimization techniques? These are commonly used to explore spaces where evaluation is expensive.
They also depend on the design space to somewhat friendly in nature and can be modelled by a surrogate, so that exploit/explore can be modelled in an acquisition function.
Also successive halving e.g. build on assumptions how the learning curve develops.
Bottom line is that there is hyperparams for hyperparam searches again. So one starts building hyperparam heuristics on top of the hyperparam search.
In the end there is no free lunch. But if hyperparam search strategy somewhat works in a domain it is a great tool. Good thing is that one can typically encode the design space in Blackbox optimization algorithms more easily.
Re: Optimizing ML training with metagradient descent
#8In the ML problem I'm working on now, there are about a dozen simple hyperparameters, and each training run takes hours or even days. I don't think there's any good way to search the space of hyperparameters without a deep understanding of the problem domain, and even then I'm often surprised when a minor config tweak yields better results (or fails to). Many of these hyperparameters affect performance directly and a…
I've been wondering how the training process of the huge models works in practice. If an optimization run costs millions, they probably don't just run a grid of hyperparameters.
Re: Optimizing ML training with metagradient descent
#9In the ML problem I'm working on now, there are about a dozen simple hyperparameters, and each training run takes hours or even days. I don't think there's any good way to search the space of hyperparameters without a deep understanding of the problem domain, and even then I'm often surprised when a minor config tweak yields better results (or fails to). Many of these hyperparameters affect performance directly and a…
That's the advantage of deep learning over traditional ML: if you've got enough data, you don't need domain knowledge or hyperparameter tuning, just throw a large enough universal approximator at it. The challenge lies in generating good enough artificial data for domains without enough data, and getting deep models to perform competitively with simpler models.
Re: Optimizing ML training with metagradient descent
#10Earlier quoted context omitted.
You might find this helpful for prioritizing which knobs to turn first https://github.com/google-research/tuning_playbook
Starting to get a bit out of date. Pity they stopped updating it.
Pity indeed! Do you have any suggested resources that are more up-to-date?