AutoML toolkit for neural architecture search and hyper-parameter tuning • Hacker News Board

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#31

Post by minimaxir » Fri, Mar 01, 2019, 4:56 PM UTC

pplonski86 wrote: post #8

mmq wrote: post #5

Earlier quoted context omitted.

I think it should probably support scikit as well as any other library, since it's only making suggestions of hyper-parameters based on recorded/historical observations or random evaluations. At least that's the behaviour of the platform[1] I am working on. [1]: https://github.com/polyaxon/polyaxon#hyperparameters-tuning

I think it all depends on the purpose of the library and who is a target user. The NNI is a package for tuning neural networks models, it will be mostly used in use cases that require deep neural networks, like image classification or voice recognition. BTW, I think all autoML solutions forget about end users. They all require too much engineering knowledge from the user. I think it will be nice to have an autoML sol…

> BTW, I think all autoML solutions forget about end users. They all require too much engineering knowledge from the user. I think it will be nice to have an autoML solution that can be used by citizen data scientist.

This is the approach of a project I am currently working on. (and am now explicitly making clear in the README!)

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#32

Post by perturbation » Fri, Mar 01, 2019, 5:19 PM UTC

Their example with LightGBM (https://nni.readthedocs.io/en/latest/gbdt_example.html) is very cool - I wanted to put together a custom script with mlflow + catboost + mlrMBO to do something similar, but this puts everything together in one package.

I think this does everything MLFlow does and more (besides maybe helping with deployment?)

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#33

Post by angel_j » Fri, Mar 01, 2019, 5:34 PM UTC

Does it test against and prevent over-fitting?

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#34

Post by streetcat1 » Fri, Mar 01, 2019, 5:47 PM UTC

sandGorgon wrote: post #4

interesting - there's no scikit support, which for long has been the mainstay for data scientists everywhere. Are people migrating from scikit to tensorflow in production for non-deep learning usecases ?

scikit learn is a different type of search, hence it will not be supported by this tool or any DNN search tool.

DNN require an architecture search, I.e. the building block are full layers, depth of the network, optimizer etc.

scikit learn search a parameter space, I.e. the algorithm weight are much much simpler and few.

So to sum up, DNN search involve big building blocks while scikit learn search (or for that reason any "classical ML" algorithm) is more of a parameter search.

[ The actual sci kit learn search would also include pre processing steps, which can be seen as a separate block]

Also, note that that DNN search is much more expensive than scikit learn search (100X) ]

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#35

Post by williamsmj » Fri, Mar 01, 2019, 6:16 PM UTC

streetcat1 wrote: post #34

sandGorgon wrote: post #4

interesting - there's no scikit support, which for long has been the mainstay for data scientists everywhere. Are people migrating from scikit to tensorflow in production for non-deep learning usecases ?

scikit learn is a different type of search, hence it will not be supported by this tool or any DNN search tool. DNN require an architecture search, I.e. the building block are full layers, depth of the network, optimizer etc. scikit learn search a parameter space, I.e. the algorithm weight are much much simpler and few. So to sum up, DNN search involve big building blocks while scikit learn search (or for that reason…

This tool absolutely supports scikit-learn. Please see the docs. https://nni.readthedocs.io/en/latest/sklearn_examples.html.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#36

Post by p1esk » Fri, Mar 01, 2019, 6:27 PM UTC

bitforger wrote: post #25

bitL wrote: post #18

Earlier quoted context omitted.

Google, Facebook & MS already have even automated research, i.e. automated selection of a loss function, network architecture, individualized network topology etc. Amazon is not there yet. The rest of industry is still in "stone age", just "considering" using something like AutoML for basic hyperparameter tuning.

If you automate it, is it still research? Research implies some sort of hypothesis testing, yes? I suppose OP means there will be two groups: people who use AutoML and people who try to make AutoML better.

There should be at least 3 groups, because making AutoML better != making ML better.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#37

Post by scottlegrand2 » Fri, Mar 01, 2019, 7:22 PM UTC

sandGorgon wrote: post #4

interesting - there's no scikit support, which for long has been the mainstay for data scientists everywhere. Are people migrating from scikit to tensorflow in production for non-deep learning usecases ?

At a previous gig we tried to do this: port a computational graph that wasn't a neural network to tensorflow. It was a disaster. Tensorflow is very tightly optimized for the things Google think are important. if you fall off of those paths tensorflow is a god-awful slow tool to use. We saw a ~20x regression in performance.

in contrast, when we wrote bespoke GPU code for the graph, we saw a ~25x performance increase over relying on CPU plus MKL. I am being deliberately vague here and I cannot give further detail.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#38

Post by human_scientist » Fri, Mar 01, 2019, 7:25 PM UTC

pplonski86 wrote: post #8

mmq wrote: post #5

Earlier quoted context omitted.

I think it should probably support scikit as well as any other library, since it's only making suggestions of hyper-parameters based on recorded/historical observations or random evaluations. At least that's the behaviour of the platform[1] I am working on. [1]: https://github.com/polyaxon/polyaxon#hyperparameters-tuning

I think it all depends on the purpose of the library and who is a target user. The NNI is a package for tuning neural networks models, it will be mostly used in use cases that require deep neural networks, like image classification or voice recognition. BTW, I think all autoML solutions forget about end users. They all require too much engineering knowledge from the user. I think it will be nice to have an autoML sol…

What about approaches like auto-sklearn [1]? With these it is basicaly:

  >>> automl = autosklearn.classification.AutoSklearnClassifier()
  >>> automl.fit(X_train, y_train)
  >>> y_hat = automl.predict(X_test)

[1] https://automl.github.io/auto-sklearn/stable/

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#39

Post by human_scientist » Fri, Mar 01, 2019, 7:36 PM UTC

streetcat1 wrote: post #34

sandGorgon wrote: post #4

interesting - there's no scikit support, which for long has been the mainstay for data scientists everywhere. Are people migrating from scikit to tensorflow in production for non-deep learning usecases ?

scikit learn is a different type of search, hence it will not be supported by this tool or any DNN search tool. DNN require an architecture search, I.e. the building block are full layers, depth of the network, optimizer etc. scikit learn search a parameter space, I.e. the algorithm weight are much much simpler and few. So to sum up, DNN search involve big building blocks while scikit learn search (or for that reason…

Automatically building a scikit learn estimator might include many conditional hyperparameters and also a very large amount of them (The tools included in the repository are very broadly applicable and only a few of them are specifically targeted at neural architecture search.

[1] https://www.kdnuggets.com/2016/08/winning-automl-challenge-a... [2] https://openreview.net/forum?id=ByfyHh05tQ

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#40

Post by human_scientist » Fri, Mar 01, 2019, 7:40 PM UTC

mlthoughts2018 wrote: post #11

mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

The parent did not specifically talk about NNs. As I understand it AutoML could apply to all statistical endeavours that involve estimation (classical or bayesian).