Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning Done Wrong

ml.posthaven.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Machine Learning Done Wrong

#11

Post by orting »

I think the points are good, but I am not very happy about this statement

"When dealing with small amounts of data, it’s reasonable to try as many algorithms as possible and to pick the best one since the cost of experimentation is low. But as we hit “big data”, it pays off to analyze the data upfront and then design the modeling pipeline (pre-processing, modeling, optimization algorithm, evaluation, productionization) accordingly."

If done correctly, then I agree. But we have to be carefull about overfitting when we try out several models or make an initial analysis to determine which model to use. In this sense, choosing a model is no different from fitting the parameters of the model.

Re: Machine Learning Done Wrong

#12

Post by idunning »

orting wrote: post #11

I think the points are good, but I am not very happy about this statement "When dealing with small amounts of data, it’s reasonable to try as many algorithms as possible and to pick the best one since the cost of experimentation is low. But as we hit “big data”, it pays off to analyze the data upfront and then design the modeling pipeline (pre-processing, modeling, optimization algorithm, evaluation, productionizatio…

If you are disciplined, and separate data into training and testing sets, you can try as many models as you want without fear of overfitting. Indeed, optimizing over the parameters of a model on the training set is essential (pruning parameters in a tree, regularization weights, etc.) and can be thought of as training large number of models.

If you aren't doing this correctly, then you can't really interpret the performance of even a single model. Seen people screw this up in so many ways - my favorite recent one that was quite high on HN was someone using the full dataset for variable selection, before doing a training-testing split afterwards.

Re: Machine Learning Done Wrong

#13

Post by idunning »

mystique wrote: post #10

Good list. I am new to Machine Learning with only ~1 year of real work and sometimes I slip and make one of these mistakes. I have a question on #7. I have not used the co-efficients to mean feature importance but some times get tempted to use them. How do you explain which factors are the most important factors behind some outcome to non-stat people?

Some techniques, e.g. random forest, give variable importance indicators for free. If you can test it out, give it a go - don't have to use the random forest as the final model.

Re: Machine Learning Done Wrong

#14

Post by mdbco »

mystique wrote: post #10

Good list. I am new to Machine Learning with only ~1 year of real work and sometimes I slip and make one of these mistakes. I have a question on #7. I have not used the co-efficients to mean feature importance but some times get tempted to use them. How do you explain which factors are the most important factors behind some outcome to non-stat people?

Point #7 is just referring to the magnitudes (or absolute values) of the coefficients. You can still determine which features are relatively important using the coefficient p-values if those are available. This of course is dependent on the necessary assumptions of the regression method that you are using being satisfied, as otherwise the p-values will be biased.

In terms of explaining this to non-stats people, you might want to avoid explaining the p-values directly to them (as it's very easy for people to get confused about what p-values actually mean), so instead you might simply show them which features are "statistically significant". In other words, try to explain the results in a qualitative way rather than a strictly quantitative one.

Re: Machine Learning Done Wrong

#15

Post by ma2rten »

idunning wrote: post #13
mystique wrote: post #10

Good list. I am new to Machine Learning with only ~1 year of real work and sometimes I slip and make one of these mistakes. I have a question on #7. I have not used the co-efficients to mean feature importance but some times get tempted to use them. How do you explain which factors are the most important factors behind some outcome to non-stat people?

Some techniques, e.g. random forest, give variable importance indicators for free. If you can test it out, give it a go - don't have to use the random forest as the final model.

You can use correlation or the coefficients of a linear model iff the features on the same scale. Another method is to train a model leaving out each feature once, then you see how much accuracy drops.

Re: Machine Learning Done Wrong

#16

Post by ma2rten »

Returning to fraud detection, high order interaction features like "billing address = shipping address and transaction amount

I agree, that non-linear models are often able to beat linear ones, but if you have limited amounts of data feature engineering will always beat clever algorithms.

Re: Machine Learning Done Wrong

#17

Post by chengtao »

antonb2011 wrote: post #8

You suggest up/down sampling rare cases. Can you please elaborate on the standard approaches for this kind of problem? For both linear and nonlinear classifiers. Thank you.

Great question and my main point is less about up sampling the rare cases but more about the default loss function used in the model training might not directly align with the final business metric (which is the metric practitioners should care more about). As a result, it's important to align the both. For some algorithms, it's easier to incorporate different loss function, while for some others, it might not be the case. Over or under sampling is one fairly generally applicable way to tweak the loss function.

While I'm not an expert of the theory behind sampling, if you do find the need to tweak sampling to align the default loss function and the business metric, I would say doing grid search first, and validate the result with the business insight, e.g. if you find getting the rare cases right is much more important that getting the common cases right, does that align with the business insight?)

Re: Machine Learning Done Wrong

#18

Post by mdbco »

"Statistical modeling is a lot like engineering."

I can certainly see why this is a good comparison, because it's true that both engineering methods and statistical methods rely on sets of given assumptions, but it's also really important not to take this analogy too far. Engineering is ultimately something that is done in a mechanistic world with primarily deterministic outcomes, whereas statistical modeling is conducted in a stochastic world with probabilistic outcomes, so it wouldn't be good to think about machine learning as predominantly mechanistic in nature (in spite of its name). Of course, a lot of the seven points that follow in the post actually emphasize the importance of stochastic factors (e.g. outliers, variance issues, collinearity, etc), so the author is clearly not making this mistake, but it might be good to clarify for anyone else who is reading.

"6. Use linear model without considering multi-collinear predictors"

This is a great point, and just to expand on it a bit, you can also have situations where you have simultaneity, i.e. two or more of your features or predictors are either functions of each other and/or functions of some third variable. This type of problem is more difficult to detect but can cause serious problems with interpreting the regression coefficients as it's ultimately a type of endogeneity, which means that common approaches like OLS will not be consistent.

Re: Machine Learning Done Wrong

#19

Post by chengtao »

mystique wrote: post #10

Good list. I am new to Machine Learning with only ~1 year of real work and sometimes I slip and make one of these mistakes. I have a question on #7. I have not used the co-efficients to mean feature importance but some times get tempted to use them. How do you explain which factors are the most important factors behind some outcome to non-stat people?

All the other comments are great. Just bear in mind that it's important to really understand the mechanics behind each importance measurement. Some can use information gain, some can use the t-test on coefficient, while some use random forest and see if removing a feature makes big impact, etc. They all make different assumptions and the key point is again, understand whether those assumptions applied to your situation.

Re: Machine Learning Done Wrong

#20

Post by chengtao »

orting wrote: post #11

I think the points are good, but I am not very happy about this statement "When dealing with small amounts of data, it’s reasonable to try as many algorithms as possible and to pick the best one since the cost of experimentation is low. But as we hit “big data”, it pays off to analyze the data upfront and then design the modeling pipeline (pre-processing, modeling, optimization algorithm, evaluation, productionizatio…

I personally love the topic of bayesian optimization over all the possible parameters including model choice. My point was more about given the resource is always constrained, it typically pays off long term for practitioners to analyze the data, understand the underlying mechanics before jumping into modeling.
Post reply on HN