Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AutoML toolkit for neural architecture search and hyper-parameter tuning

github.com

Post reply on HN
Print view View original

21–30 of 60 posts

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#21

Post by byebyetech »

mlthoughts2018 wrote: post #11
mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

Deep Learning also works on very small data sets by means of embeddings. A large model trained on large data sets can be used as feature extraction tool for training for small data sets.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#22

Post by samcodes »

cuchoi wrote: post #16
sandGorgon wrote: post #4

interesting - there's no scikit support, which for long has been the mainstay for data scientists everywhere. Are people migrating from scikit to tensorflow in production for non-deep learning usecases ?

I think that for Neural Networks scikit has not been the "go to" library, in particular AutoML advertises that they automate neural architecture search which I don't think scikit allows a lot of flexibility for that.

Have you seen TPOT [0]? AutoML library that uses genetic algorithms to write scikit code for you. So fun.

[0] https://github.com/EpistasisLab/tpot

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#23

Post by mlthoughts2018 »

pplonski86 wrote: post #19
DebtDeflation wrote: post #15

Earlier quoted context omitted.

It's another example of the FAANG + Bay Area Startups world versus the other 99% of Corporate America. In the latter world, most of the "machine learning" in production is traditional stuff like Random Forest, SVM, and more recently Gradient Boosting. Hell, Marketing departments across the country are still running old school decision tree (CART and CHAID) models and logistic regression models written in SAS 20+ year…

I think there is good reason that "old" machine learning models are more popular than DNN in the enerprise space. Most of the data is in the tabular format. What is more, "old" and simple decision tree or linear model are very easy to understand, deploy and are fast. There is for sure clear advantage of having even simple decision tree implemented in the system than making decisions at random.

The main reason though is that these other methods outperform neural nets in tons of different situations. Even just from an accuracy / business success metric point of view, many problems are just better solved with other classes of models, domain-specific feature engineering, etc. It will probably remain so for many decades at least.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#24

Post by paperwork »

byebyetech wrote: post #21
mlthoughts2018 wrote: post #11

Earlier quoted context omitted.

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

Deep Learning also works on very small data sets by means of embeddings. A large model trained on large data sets can be used as feature extraction tool for training for small data sets.

I’ve seen this mentioned before, including a blog post by the fast.ai folks. Any idea where I can get details? If my tabular data set is small, what kind of embedding can I get out of it? Or is the idea that a larger data set is used for embeddings of categorical data?

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#25

Post by bitforger »

bitL wrote: post #18
mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

Google, Facebook & MS already have even automated research, i.e. automated selection of a loss function, network architecture, individualized network topology etc. Amazon is not there yet. The rest of industry is still in "stone age", just "considering" using something like AutoML for basic hyperparameter tuning.

If you automate it, is it still research? Research implies some sort of hypothesis testing, yes?

I suppose OP means there will be two groups: people who use AutoML and people who try to make AutoML better.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#26

Post by sdenton4 »

paperwork wrote: post #24
byebyetech wrote: post #21

Earlier quoted context omitted.

Deep Learning also works on very small data sets by means of embeddings. A large model trained on large data sets can be used as feature extraction tool for training for small data sets.

I’ve seen this mentioned before, including a blog post by the fast.ai folks. Any idea where I can get details? If my tabular data set is small, what kind of embedding can I get out of it? Or is the idea that a larger data set is used for embeddings of categorical data?

I think this is mainly a thing for perception (images and sounds). Tabular data would have to match up with the training dataset, and "most" interesting tabular models are the sports of things guarded like piles of gold by the businesses that build them...

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#27

Post by yorwba »

paperwork wrote: post #24
byebyetech wrote: post #21

Earlier quoted context omitted.

Deep Learning also works on very small data sets by means of embeddings. A large model trained on large data sets can be used as feature extraction tool for training for small data sets.

I’ve seen this mentioned before, including a blog post by the fast.ai folks. Any idea where I can get details? If my tabular data set is small, what kind of embedding can I get out of it? Or is the idea that a larger data set is used for embeddings of categorical data?

Pre-trained embeddings are only helpful if they are trained on a different (ideally larger) dataset or even a different task, but with the same kind of input data. So you would need to find out where else something similar to the data in your tables appears. If some of the data is text, word embeddings may be applicable. Or if you're trying to analyze user activity by time and location, you might try to transfer what can be learned about the influence of holidays from publicly observable activity e.g. on Twitter (just a random idea that popped into my head, no guarantee that it can actually work).

Of course if all you have are numbers without context, there isn't a lot you can do to improve the situation.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#28

Post by psandersen »

mlthoughts2018 wrote: post #23
pplonski86 wrote: post #19

Earlier quoted context omitted.

I think there is good reason that "old" machine learning models are more popular than DNN in the enerprise space. Most of the data is in the tabular format. What is more, "old" and simple decision tree or linear model are very easy to understand, deploy and are fast. There is for sure clear advantage of having even simple decision tree implemented in the system than making decisions at random.

The main reason though is that these other methods outperform neural nets in tons of different situations. Even just from an accuracy / business success metric point of view, many problems are just better solved with other classes of models, domain-specific feature engineering, etc. It will probably remain so for many decades at least.

DNN's make good features though, especially if you have time series data or lots of text.

I agree that the final model should be a randomforest/xgboost/lightgbm for typical tabular data.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#29

Post by williamsmj »

I'd be interested in the creator's thoughts on this paper, "Random Search and Reproducibility for Neural Architecture Search", https://arxiv.org/abs/1902.07638, posted on the arxiv last week. Among other conclusions, they find:

"Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks"

ENAS, the specific algorithm that they find does no better than chance, is in this library. My understanding is that the results are pretty generic though, i.e. NAS is very far from a solved problem. (Hyperparameter tuning for "classical" models are another matter. That's commoditized and available as a service at this point, see tpot, DataRobot, etc., etc.)

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#30

Post by williamsmj »

sandGorgon wrote: post #4

interesting - there's no scikit support, which for long has been the mainstay for data scientists everywhere. Are people migrating from scikit to tensorflow in production for non-deep learning usecases ?

There is scikit support. There's an example in the docs.

https://nni.readthedocs.io/en/latest/sklearn_examples.html

Post reply on HN