Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AutoML toolkit for neural architecture search and hyper-parameter tuning

github.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#41

Post by mlthoughts2018 »

psandersen wrote: post #28
mlthoughts2018 wrote: post #23

Earlier quoted context omitted.

The main reason though is that these other methods outperform neural nets in tons of different situations. Even just from an accuracy / business success metric point of view, many problems are just better solved with other classes of models, domain-specific feature engineering, etc. It will probably remain so for many decades at least.

DNN's make good features though, especially if you have time series data or lots of text. I agree that the final model should be a randomforest/xgboost/lightgbm for typical tabular data.

I meant that extracting an intermediate layer as a feature embedding and then sticking a classical model on top of it performs worse than curating features through domain-specific expert tuning, for a ton of diverse application domains.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#42

Post by mlthoughts2018 »

byebyetech wrote: post #21
mlthoughts2018 wrote: post #11

Earlier quoted context omitted.

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

Deep Learning also works on very small data sets by means of embeddings. A large model trained on large data sets can be used as feature extraction tool for training for small data sets.

Re-using an existing model to generate embeddings doesn’t work well for auxiliary tasks with very small data. Even if you do no fine-tuning at all, you need to have big data sets in terms of the auxiliary task too.

For example, consider needing to train hundreds of unique small models every day, based on new customer inputs affecting causality effects for that day (I had to do this for ad forecasting in a past job).

Generating embeddings via pre-trained models essentially produced gibberish and performed far worse than custom feature engineering + simple logistic models.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#43

Post by mlthoughts2018 »

human_scientist wrote: post #40
mlthoughts2018 wrote: post #11

Earlier quoted context omitted.

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

The parent did not specifically talk about NNs. As I understand it AutoML could apply to all statistical endeavours that involve estimation (classical or bayesian).

> “AutoML could apply to all statistical endeavours that involve estimation”

Yes, this is the part that sounds like parody to me. At least, as a working statistician, I can tell you that the concept of AutoML could not apply to the far majority of things I work on.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#44

Post by yorwba »

mlthoughts2018 wrote: post #43
human_scientist wrote: post #40

Earlier quoted context omitted.

The parent did not specifically talk about NNs. As I understand it AutoML could apply to all statistical endeavours that involve estimation (classical or bayesian).

> “AutoML could apply to all statistical endeavours that involve estimation” Yes, this is the part that sounds like parody to me. At least, as a working statistician, I can tell you that the concept of AutoML could not apply to the far majority of things I work on.

So you never tune hyperparameters or try different models to see which works better?

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#45

Post by mlthoughts2018 »

yorwba wrote: post #44
mlthoughts2018 wrote: post #43

Earlier quoted context omitted.

> “AutoML could apply to all statistical endeavours that involve estimation” Yes, this is the part that sounds like parody to me. At least, as a working statistician, I can tell you that the concept of AutoML could not apply to the far majority of things I work on.

So you never tune hyperparameters or try different models to see which works better?

I do plenty of that, and AutoML could help with a small fraction of that.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#46

Post by human_scientist »

mlthoughts2018 wrote: post #43
human_scientist wrote: post #40

Earlier quoted context omitted.

The parent did not specifically talk about NNs. As I understand it AutoML could apply to all statistical endeavours that involve estimation (classical or bayesian).

> “AutoML could apply to all statistical endeavours that involve estimation” Yes, this is the part that sounds like parody to me. At least, as a working statistician, I can tell you that the concept of AutoML could not apply to the far majority of things I work on.

Could you give an example? I have a hard time understanding what you could mean, as Algorithm Configuration & Selection is such a general framework. If you are solely talking about the current state of the art, I would agree that techniques from AutoML do not have the generality and autonomity of an expert human.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#47

Post by human_scientist »

p1esk wrote: post #36
bitforger wrote: post #25

Earlier quoted context omitted.

If you automate it, is it still research? Research implies some sort of hypothesis testing, yes? I suppose OP means there will be two groups: people who use AutoML and people who try to make AutoML better.

There should be at least 3 groups, because making AutoML better != making ML better.

Why? The concept of AutoML does include the design of novel algorithms.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#48

Post by noelsusman »

mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

Hasn't this always been the case? Actually fitting a model has always been a pretty small part of an applied statistician's job. The real work is everything before and after that point.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#49

Post by yzh »

I'm working on auto hyper-parameter tuning and network optimization, I always think that people have put too much focus on NAS, which aims to create a whole new network from scratch, but not nearly enough on hyper-parameter tuning and local structural optimizations for an existing network, which I think is more demanding at least in the industry. Looks less cool than NAS though, maybe that's the reason.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#50

Post by p1esk »

human_scientist wrote: post #47
p1esk wrote: post #36

Earlier quoted context omitted.

There should be at least 3 groups, because making AutoML better != making ML better.

Why? The concept of AutoML does include the design of novel algorithms.

What do you mean? I thought AutoML was a tool to do neural architecture search, and hyperparameter tuning.
Post reply on HN