Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AutoML toolkit for neural architecture search and hyper-parameter tuning

github.com

Post reply on HN
Print view View original

11–20 of 60 posts

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#11

Post by mlthoughts2018 »

mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg).

In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridiculous thing to try.

I think the breakdown of workload you described will only happen in business departments where there is a need for large scale embedding models, enhanced multi-modal search indices, computer vision and natural language applications, and maybe a handful of things that eventually productize reinforcement learning. I could also see this happening in businesses that can benefit from synthetically generated content, like stock photography, essays / news summaries / some fiction, website generators, probably more.

What I described above is a tiny drop in the ocean of applied statistics problems that business have to solve.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#12

Post by mjburgess »

mlthoughts2018 wrote: post #11
mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

The problem is "Applied Statistics" became "Machine Learning" which became "AI" which became "Deep Learning".

Throw away all the BS. and, yes, it's obvious.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#15

Post by DebtDeflation »

mlthoughts2018 wrote: post #11
mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

It's another example of the FAANG + Bay Area Startups world versus the other 99% of Corporate America. In the latter world, most of the "machine learning" in production is traditional stuff like Random Forest, SVM, and more recently Gradient Boosting. Hell, Marketing departments across the country are still running old school decision tree (CART and CHAID) models and logistic regression models written in SAS 20+ years ago. DL/NN is a minuscule proportion of production ML in the enterprise space.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#16

Post by cuchoi »

sandGorgon wrote: post #4

interesting - there's no scikit support, which for long has been the mainstay for data scientists everywhere. Are people migrating from scikit to tensorflow in production for non-deep learning usecases ?

I think that for Neural Networks scikit has not been the "go to" library, in particular AutoML advertises that they automate neural architecture search which I don't think scikit allows a lot of flexibility for that.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#17

Post by glial »

sgt101 wrote: post #14

I don't understand - isn't this model fishing? How is it different?

Yes, but that's not necessarily bad. You want a model that effectively captures the structure present in your dataset. There are currently only rules-of-thumb in model architecture, and it makes sense to explore the model space to determine which architecture and hyper parameters are suitable to the needs at hand. Two things save this from being a statistical sin: one, the final evaluation set is typically different than the validation set, and evaluation is only performed at the end of the 'fishing expedition', thus providing a reliable measure of the model's ability to generalize. Second, we're doing engineering here, not science, and our goal is to capture the structure of observations and not make a scientific claim about values of latent parameters.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#18

Post by bitL »

mark_l_watson wrote: post #10

I manage a machine learning team for a large financial services company and AutoML tools, Microsoft’s NNI included, are on our radar. I think the `future of work` for machine learning practitioners will quickly separate into two groups: a very small and elite group that performs research and a much larger groups that use AutoML but whose jobs also deal more with data preparation (which gets automated also) and ML dev…

Google, Facebook & MS already have even automated research, i.e. automated selection of a loss function, network architecture, individualized network topology etc. Amazon is not there yet. The rest of industry is still in "stone age", just "considering" using something like AutoML for basic hyperparameter tuning.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#19

Post by pplonski86 »

DebtDeflation wrote: post #15
mlthoughts2018 wrote: post #11

Earlier quoted context omitted.

This sounds like parody to me. There are so many problems in applied statistics, and neural networks are not helpful for most of them. Consider Bayesian analysis for very small data sets as an example (just the tip of the iceberg). In financial services in particular, there are tons of time series and regression problems on small data such that a neural network (beyond perhaps some super small MLP) would be a ridicul…

It's another example of the FAANG + Bay Area Startups world versus the other 99% of Corporate America. In the latter world, most of the "machine learning" in production is traditional stuff like Random Forest, SVM, and more recently Gradient Boosting. Hell, Marketing departments across the country are still running old school decision tree (CART and CHAID) models and logistic regression models written in SAS 20+ year…

I think there is good reason that "old" machine learning models are more popular than DNN in the enerprise space. Most of the data is in the tabular format. What is more, "old" and simple decision tree or linear model are very easy to understand, deploy and are fast. There is for sure clear advantage of having even simple decision tree implemented in the system than making decisions at random.

Re: AutoML toolkit for neural architecture search and hyper-parameter tuning

#20

Post by thanatropism »

sgt101 wrote: post #14

I don't understand - isn't this model fishing? How is it different?

With training, test and validation sets.

In good old fashioned statistics there's the idea of the jackknife: for the i-th sample run a regression on all the data except i, and store statistics of interest (coefficients, predictions, etc). This gives you an ipso facto sampling distribution for the statistics of interest.

Similar and more common in econometrics is the bootstrap: run your model in like 1999 subsamples (with repetition) of the data and get sampling distributions.

With said sampling distributions, whether from the jackknife or the bootstrap, you're able to test whether your model is valid -- what's the probability that it'll have significant coefficients or an r2/mae/mape score indicating predictive capacity.

Cross-validation (and even scikit-learn is starting to default to five folds not three) is a "lazy" version of this. You don't get a sampling distribution but at least you're able to know that a given model appears good because it grips the data with all its might and doesn't work out-of-sample.

sklearn even offers the jackknife under some ML-y name like "one at a time scoring".

Post reply on HN