Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An End-to-End AutoML Solution for Tabular Data at KaggleDays

ai.googleblog.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#21

Post by so_tired »

fnbr wrote: post #3
rahimnathwani wrote: post #2

I'm interested to know how easy this for regular people (software engineers with just a little knowledge of data science) to use. This part stands out: "our team spent most of time monitoring jobs and and waiting for them to finish. Our solution for second place on the final leaderboard required 1 hour on 2500 CPUs" Before I got to this part, I had assumed using AutoML would involve only reformatting the training/val…

(I'm a researcher in ML/DL.) > Anyone know why they used CPUs instead of GPUs/TPUs? Cost and resource availability. They're distributing each computation on a different CPU, not distributing each computation over multiple CPUs. It would be faster to have each computation run on a CPU + GPU, but that would be very very expensive, and hard to schedule. GPUs/TPUs are also only faster for sufficiently large networks and…

Is there any way to "scale up" the solution network for more dimensions?

For example, if i train on a a 100x100 image domain, with good results. So now I get a bigger budget to work with 200x200 images. There is no real way to leverage the good architecture from the 1st network. Is there ?

Can this be done as a ugly-hack and then be used as a seed into the architecture-search ?

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#23

Post by tixocloud »

filleokus wrote: post #15

How does these auto ML-solutions (like h2o) work in practice, anyone willing to share their experience? I wonder how automatic machine learning tools like these will shape the "data science" roles in the future. Obviously, the most cutting edge research will always be done by specialised human experts, but perhaps tools like these will lower the bar required for the bulk of mainstream ML work.

Auto ML is great as long as there is some rational on why a particular model is developed. However, in real world applications, there are so many constraints and considerations needed besides just having the most accurate model.

Furthermore, as a VP of data science, selling the business on the value and benefits of data science is non-trivial as many are not even aware of what's capable with data science so my personal opinion is that ML isn't quite suitable for mainstream work. Practitioners still need to have a strong grasp of the business context, data features and nuances, etc. which still is quite technical in nature. Asking a data analyst to build models with auto ML is something I consider to be a leap too far and is risky to put into production. The only way it could work is to have an experienced data scientist supervise.

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#24

Post by mlthoughts2018 »

Tenoke wrote: post #17
ipsum2 wrote: post #8

> Erkut Aykutlug and Mark Peng used XGBoost with creative feature engineering whereas AutoML uses both neural network and gradient boosting tree (TFBT) with automatic feature engineering and hyperparameter tuning. It's hilarious that gradient boosted descent tree beat Google's fancy AutoML-generated neural networks.

Im guessing 'creative feature engineering' vs 'automatic' suggests people were involved in it versus the completely end to end approach by AutoML. Im guessing that's the main reason they beat Google.

That’s exactly what’s _not_ supposed to perform better than AutoML...

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#25

Post by mlthoughts2018 »

tixocloud wrote: post #23
filleokus wrote: post #15

How does these auto ML-solutions (like h2o) work in practice, anyone willing to share their experience? I wonder how automatic machine learning tools like these will shape the "data science" roles in the future. Obviously, the most cutting edge research will always be done by specialised human experts, but perhaps tools like these will lower the bar required for the bulk of mainstream ML work.

Auto ML is great as long as there is some rational on why a particular model is developed. However, in real world applications, there are so many constraints and considerations needed besides just having the most accurate model. Furthermore, as a VP of data science, selling the business on the value and benefits of data science is non-trivial as many are not even aware of what's capable with data science so my person…

Exactly this. The combination of selling a modeling strategy to higher ups, customizing a model for super weird deployment or resource limitations that exist for political or historical reasons, and all your standard trade-offs vs feature engineering and model selection just means that AutoML (and similar tools) has very little applicability in most product companies.

If you work in Google & you can remove the political blockers & guarantee the model’s space of deployed resources is parametrizable with a clean set of parameters that AutoML can consider as part of the optimization, then by all means use it.

That’s just decades away from being viable at any given product company.

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#26

Post by pplonski86 »

Tarq0n wrote: post #14
ptah wrote: post #10

Can automl be replicated outside of google cloud?

There are many autoML libraries, however I'm not aware off any that do production-grade neural architecture search.

I'm working on AutoML solution, that is available in the cloud (https://mljar.com). What is more, the core of my AutoML is open source (https://github.com/mljar/mljar-supervised) - both are easy to use. The cloud version has user interface so you need just to upload data and do few clicks. You don't need to have programming knowledge. For python package, you need to know how to manipulate data in python (basic numpy and pandas).

I've made many tests of my AutoML solution and I observe that Neural Networks doesn't work the best on tabular dataset (maybe not training long enough, but I dont have 2500CPU hours). I really prefer gradient boosting methods (xgboost, catboost, lightgbm) on tabular data. They are much faster than NN and require less preprocessing (no feature scaling).

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#28

Post by pplonski86 »

ptah wrote: post #9
ipsum2 wrote: post #8

> Erkut Aykutlug and Mark Peng used XGBoost with creative feature engineering whereas AutoML uses both neural network and gradient boosting tree (TFBT) with automatic feature engineering and hyperparameter tuning. It's hilarious that gradient boosted descent tree beat Google's fancy AutoML-generated neural networks.

yes, and i bet they didn't use 2500 CPUs either

The first thing is, the Neural Networks are not golden ML algorithm that is working always the best. I'm not surprised to see GBDT methods working better than NN. The NNs are very powerful because they can accept the wide-range of data types (tabular, images, voice) which is not possible with GBDT (maybe not yet).

The second, I think architecture search fo NNs is very inefficient right now. Most of the methods are training new NN in every attempt. There is a lack for method that can start with any NN architecture and grow/prune it dynamically. Take a look at this poster, with dynamic topology adjustemnt for MLP http://www.ire.pw.edu.pl/~rsulej/NetMaker/icaisc/icaisc_post...

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#29

Post by kelvin0 »

Let's just hope this does not become the 'Excel' of the ML space. Then anyone will start 'coding' some godawful models and use them in critical day to day infrastructure ...

Don't get me wrong, I'm all for democratizing ML, but sometime these tools become fully-automatic-high-caliber footguns.

Re: An End-to-End AutoML Solution for Tabular Data at KaggleDays

#30

Post by bouk »

kelvin0 wrote: post #29

Let's just hope this does not become the 'Excel' of the ML space. Then anyone will start 'coding' some godawful models and use them in critical day to day infrastructure ... Don't get me wrong, I'm all for democratizing ML, but sometime these tools become fully-automatic-high-caliber footguns.

Excel is an amazing tool, and if this becomes the Excel of the ML space then it will be a great success
Post reply on HN