Live data from Hacker News

What’s the difference between statistics and machine learning?

thestatsgeek.com

21–30 of 95 posts

Re: What’s the difference between statistics and machine learning?

#21
The classic explanation is Lei Breiman's 'Two Cultures' paper. He was a statistics professor who left for industry, came back, and tried to get academics to adapt industry approaches. The paper is very readable.

http://www2.math.uu.se/~thulin/mm/breiman.pdf

An oversimplified version may be:

Statistics focuses on fitting data to formally understandable models, whereas data science focuses on solving problems -- even if that means using techniques that aren't formally understood.

Leo Breiman is also known for pioneering random forest and bootstrap aggregation.

I think of machine learning as a subset of data science where you trick linear algebra into thinking.

Re: What’s the difference between statistics and machine learning?

#23
post #18
post #11

That's easy - statisticians take pride in models that are understandable, while machine learning practitioners take pride in models that are not.

I'd change that to say machine learning practitioners take pride in performance. If the best method of predicting planetary motions is to model them as a hierarchy of triangular epicycles then that's exactly what a machine learning practitioner will do.

That's kind of my point, that they have different tradeoffs, although I do not hide my preference for explainability (or model soundness). :-) The example you give reminds me of Chomsky and Norvig debate on the topic.

Re: What’s the difference between statistics and machine learning?

#24
Statistics is good for modelling things that are simplistic & low dimensional. Machine learning is good for modelling things that are nuanced & high dimensional. statisticians want to understand things but imho overestimate human ability to make sense of a complex world. machine learning people want the machine to understand the things for us and then teach us about it when it's making us breakfast.

Re: What’s the difference between statistics and machine learning?

#25

This may sound a little like trivializing, but don't we have to know what "statistics" are and what "machine learning" is to say anything about the difference(s) between them? Looking at this through even the lens of multinomial logistic regression, or of econometrics generally, I don't think that "statistics draws population inferences from a sample, while machine learning finds generalizable predictive patterns" ev…

Have fun trying to model extremely nonlinear physical effects with statistics

Oh yes, we just include all the possible nonlinear effects and pick the ones which look meaningful [0][1]

Then we go back and perturb the model to do some hand wavey guessing at real actionable insights

but hey clients are impressed by overfitting so who the fuck cares as long as the money is coming in and the press releases are going out!!!!

[0] https://www.featuretools.com/ [1] https://scikit-learn.org/stable/modules/generated/sklearn.pr...

Re: What’s the difference between statistics and machine learning?

#27

The classic explanation is Lei Breiman's 'Two Cultures' paper. He was a statistics professor who left for industry, came back, and tried to get academics to adapt industry approaches. The paper is very readable. http://www2.math.uu.se/~thulin/mm/breiman.pdf An oversimplified version may be: Statistics focuses on fitting data to formally understandable models, whereas data science focuses on solving problems -- even i…

[deleted]

Re: What’s the difference between statistics and machine learning?

#28
Inferential statistics is about explaining an observed outcome in terms of its causing factors. Once we have explained it, then we can make predictions. Machine learning skips the explaining part and goes straight to making predictions, without attempting to understand the underlying process that led to the particular outcome. This would be the main difference, in my opinion.

Re: What’s the difference between statistics and machine learning?

#29
post #14
post #9

Earlier quoted context omitted.

Everybody owes it to themselves to read Breiman's Two Cultures[1]: > There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has le…

Nice comment, but there's more nuance than you claim. > You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. You made implicit assumptions that you are now unaware of, which might come and bite you later (eg: using zip codes or names as a proxy for race in credit scoring models, neural networks overfititng to texture and classifying a leopard print couch as…

I think what you're saying is orthogonal to what I'm saying.

Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem you're stating.

What you have not made any assumptions about in a random forest is about the distribution of the data you're looking at. One example of a case where the assumptions that bog-standard OLS makes about your data can cause you problems is zero-dominated data -- data with a lot of zeros in it. Basically any time you're trying to make predictions about things that are rare in your measured population.

OLS does a bad job on zero-dominated data. If you throw a zero-dominated dataset into a random forest, you will get back better answers than if you use OLS on zero-dominated data.

To be clear: there are strategies for dealing with zero-dominated data using statistical inference. You don't have to resort to non-inferential learning just because you have data that doesn't look like a bell curve. But machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved (or in cases where the probability function is too complicated to be tractable computationally, like the probability function that determines the color of pixels in a dataset where you're classifying dogs versus cats).

Post reply on HN