Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning Done Wrong

ml.posthaven.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Machine Learning Done Wrong

#21

Post by chengtao »

ma2rten wrote: post #16

Returning to fraud detection, high order interaction features like "billing address = shipping address and transaction amount I agree, that non-linear models are often able to beat linear ones, but if you have limited amounts of data feature engineering will always beat clever algorithms.

Yes, and IMO, most of the time, the insight behind the data is far more important than the modeling algorithms to achieve high performance with few exceptions (say computer vision, NLP, etc which really requires A LOT OF data). Even in some large data set, take page rank as an example. The fundamental insight was the popularity of the site would be a great signal for ranking the search result, and random walk would be a great way to approximate the popularity. As a result, Google made a great success in search ranking.

Re: Machine Learning Done Wrong

#22

Post by chengtao »

mdbco wrote: post #18

"Statistical modeling is a lot like engineering." I can certainly see why this is a good comparison, because it's true that both engineering methods and statistical methods rely on sets of given assumptions, but it's also really important not to take this analogy too far. Engineering is ultimately something that is done in a mechanistic world with primarily deterministic outcomes, whereas statistical modeling is cond…

great comment and +1

Re: Machine Learning Done Wrong

#23

Post by kimolas »

trunkation wrote: post #6
chengtao wrote: post #3

Earlier quoted context omitted.

Not really, it was actually more inspired by Statistics Done Wrong.

Oh, Ok. Never knew there was a book by that title.

It was written by someone in my PhD cohort—it's available for preorder now. http://www.statisticsdonewrong.com

Re: Machine Learning Done Wrong

#24

Post by karthikv2k »

"2. Use plain linear models for non-linear interaction" It should be noted that Linear models are only linear in the model parameters, while the features can be transformed using non-linear functions. This trick makes linear models very powerful. Also if you have big data (in millions/billions) then you are better off with linear models, as SVM is very difficult to scale.

In my experience (all in big data), I rarely seen people use SVM, usual choices are logistic regressions and tree based models. In some finance and insurance industries you are restricted to use only interpretable models, which linear models are.

Re: Machine Learning Done Wrong

#25

Post by dustintran »

I strongly disagree with not using linear models, at least to build some theory and intuition before continuing with more sophisticated algorithms. What I find to be more egregiously misused when doing machine learning in practice is that everyone too often flocks to the state of the art with little understanding why. There's no reason for example to spend weeks (or months) tuning a incredibly deep neural network if the current predictive ability is enough and there are higher priority matters to work on.

Moreover, there's just too much of an emphasis on prediction. Design and analysis of experiments, handling missing data and the context of the data sets, and quantifying one's uncertainty about parameters in a principled manner for robust estimators are very underappreciated skills in the community. Using p values arbitrarily and "95% confidence intervals" based on an unchecked normal approximation is incredibly more harmful than not doing anything at all. There's just so much more to machine learning than supervised learning.

Re: Machine Learning Done Wrong

#26

Post by chengtao »

karthikv2k wrote: post #24

"2. Use plain linear models for non-linear interaction" It should be noted that Linear models are only linear in the model parameters, while the features can be transformed using non-linear functions. This trick makes linear models very powerful. Also if you have big data (in millions/billions) then you are better off with linear models, as SVM is very difficult to scale. In my experience (all in big data), I rarely…

As you pointed out the transforming features is powerful, I believe that's the exact reason which makes SVM powerful. Though the way features can be combined with SVM is limited, the limitation makes SVM training fast in the dual space.

On the other hand, if you wanna compare logistic regression with SVM. While the detail is pretty tricky. One simplified view is to compare linear SVM which is essentially hinge loss with L2 regularization against logistic regression with L2 regularization which is essentially negative binomial log likelihood loss with L2 regularization. If you plot the loss functions, it's easy to see how they penalize negative & positive cases differently.

Re: Machine Learning Done Wrong

#27

Post by chengtao »

dustintran wrote: post #25

I strongly disagree with not using linear models, at least to build some theory and intuition before continuing with more sophisticated algorithms. What I find to be more egregiously misused when doing machine learning in practice is that everyone too often flocks to the state of the art with little understanding why. There's no reason for example to spend weeks (or months) tuning a incredibly deep neural network if…

To add on top of that, even with great data analysis skill, I had another blog-post talking about it requires all the product, data, and engineering skills together to make a good data science team. http://ml.posthaven.com/why-building-a-data-science-team-is-...

Re: Machine Learning Done Wrong

#28

Post by stiff »

idunning wrote: post #12
orting wrote: post #11

I think the points are good, but I am not very happy about this statement "When dealing with small amounts of data, it’s reasonable to try as many algorithms as possible and to pick the best one since the cost of experimentation is low. But as we hit “big data”, it pays off to analyze the data upfront and then design the modeling pipeline (pre-processing, modeling, optimization algorithm, evaluation, productionizatio…

If you are disciplined, and separate data into training and testing sets, you can try as many models as you want without fear of overfitting. Indeed, optimizing over the parameters of a model on the training set is essential (pruning parameters in a tree, regularization weights, etc.) and can be thought of as training large number of models. If you aren't doing this correctly, then you can't really interpret the perf…

If you use performance on the test set for model selection, this is not true. It follows from simple probabilistic reasoning, the more models you try the higher the chance one will score well on both the training set and the test set by "luck", and this is especially true with small datasets. In fact it is a best practice to use a separate validation set for model selection and use the test set only for final performance evaluation, see e.g. the answer to this question:

http://stats.stackexchange.com/questions/9357/why-only-three...

Re: Machine Learning Done Wrong

#29

Post by kylebgorman »

dustintran wrote: post #25

I strongly disagree with not using linear models, at least to build some theory and intuition before continuing with more sophisticated algorithms. What I find to be more egregiously misused when doing machine learning in practice is that everyone too often flocks to the state of the art with little understanding why. There's no reason for example to spend weeks (or months) tuning a incredibly deep neural network if…

In natural language processing, we can get close to state of the art performance on nearly every major task with a linear model; usually, the feature sets contain what are essentially conjunctions of features, but these are chosen by hand, by domain experts, rather than produced with, say, a polynomial kernel.

Re: Machine Learning Done Wrong

#30

Post by texthompson »

orting wrote: post #11

I think the points are good, but I am not very happy about this statement "When dealing with small amounts of data, it’s reasonable to try as many algorithms as possible and to pick the best one since the cost of experimentation is low. But as we hit “big data”, it pays off to analyze the data upfront and then design the modeling pipeline (pre-processing, modeling, optimization algorithm, evaluation, productionizatio…

I thought exactly the same thing. Statistics is about uncertainty, and it's very easy to be misled when you don't correct for trying lots of hypotheses.
Post reply on HN