Live data from Hacker News

What’s the difference between statistics and machine learning?

thestatsgeek.com

41–50 of 95 posts

Re: What’s the difference between statistics and machine learning?

#41

Can someone correct me, if I'm wrong, but statistics doesn't allow Turing-complete models, right? Machine learning certainly allows that (for example an RNN).

This is kind of a fun question because it places the notion of “computational” computability (classical CS) alongside that of classical computability (finding a model by, say, minimizing an error which in turn boils down to calculus or linear system solution).

Long story short, because of the power of real numbers — with each one encoding an infinite sequence of bits — it’s not clear that even “simple” computations important to elementary statistical models, like exponentiating, have low computational complexity. Computing, say, “e” can be done by a Turing machine, but it’s nontrivial.

Another observation along the same lines is that some simple statistical models turn out to really complex — like “find a such that the model

  y = sin (a x)
fits sample data (x1, y1),..., where abs(y) Another way to say it may be that Turing completeness is not a very sharp tool to separate model classes.

I found this helpful: http://www.cs.cmu.edu/~lblum/PAPERS/TuringMeetsNewton.pdf

Re: What’s the difference between statistics and machine learning?

#42
post #16

The amount of hair-splitting that goes into discussing this subject is unbelievable. Clearly, they both are fairly closely intertwined - particularly given that one potential explanation boils down to the motivation of the user. So do we need an explicit taxonomy to say that one application is machine learning and another is statistics? Or does the form not matter as much as the function? Pace the OP, who I'm sure di…

I think it’s actually both important and possible to differentiate between stats and ML (and AI for that matter). Statisticians, ML engineers, and data scientists all have related but distinct skill sets, knowledge, methodological experience, and worldviews. The identification can be important because it suggests what sorts of tasks and projects an individual is well-suited for. (Notwithstanding the fact that I believe anyone can acquire skills/knowledge to be competent at any of those jobs and transition between them with a bit of effort).

For example, consider the work that the FDA does in evaluating clinical drug trial data. In this case, you need statisticians on staff that can critically analyze the statistical/methodological rigor, not to mention understand the medical domain and relevant regulations/processes. It doesn’t matter if they can only use the GUI versions of SAS and have never heard of TensorFlow, their time is better spent deeply understanding nuances of experimental design, sampling methods, causal analysis, estimation, etc. I would argue that a “good” statistician in this context might very well be “not so good” at data analysis, as long as they can clearly and accurately critique experiments and analyses and identify when things go wrong. Also consider—what would happen if you slotted a ML engineer into a role like this?

ML engineers are best at optimizing performance (not just accuracy) at some defined task, where ideally the cost of the model being wrong in some exotic way is not astronomical (e.g. approving a dangerous drug, convicting an innocent person, initiating a stock crash, corrupting the attention span of the human race). A good use case for machine learning comes from the book Pattern Recognition—sorting fish on a conveyor belt. No one gives an ounce of chum about statistical rigor in this case, and the consequences of being inaccurate are quantifiable and manageable. But building and tuning a ML system to do this at an acceptable performance level requires a ton of work. You’d need to define the sensor array / inputs, collect and label data, engineer features (more relevant before deep learning, in the case of computer vision), train and evaluate an object detection model, avoid overfitting the training data, build an anomaly detector to weed out the stray crab, ensure it works fast enough to be used in production, turbo-browse arXiv to make sure your stuff isn’t obsolete (damn, it is). This is a vastly different focus than statisticians, who wouldn’t get very far with this sort of performance-optimization problem.

Data scientists are somewhere in between a statistician and ML engineer—like a hybrid class in an RPG. They can cast a few key ML spells and have dabbled in arcane math, but can also slice up data goblins by hand (maybe even with style). With a foundation in a bit of everything, they can glue a team together and flexibly grow in multiple directions. But they can also become a waste of shiny gold if pitted against a specialized task without support from, you know, specialists.

Re: What’s the difference between statistics and machine learning?

#43
I agree that in general statistics is more concerned about inference, while machine learning focuses on prediction. On the other hand, there is such a huge overlap between the fields that it is hard to make a distinction. Also there are statistical fields which focus more on prediction the same way as ML does. For example, in geostatistics prediction is often the only goal, e.g. to predict heavy metal concentration across a domain. People accept that it is impossible to explain every bit of spatial variation and just model it by a gaussian process (the same gaussian process used in ML).

Re: What’s the difference between statistics and machine learning?

#44
post #11

That's easy - statisticians take pride in models that are understandable, while machine learning practitioners take pride in models that are not.

Understandable? Two words: statistical significance :P I've seen university employees who still don't understand what it is and can't explain it...and just about everyone who uses it gets it wrong...

Not sure why you're downvoted. Also noticed that everyone uses statistics because you're expected to, but hardly anyone knows how to do it right.

Re: What’s the difference between statistics and machine learning?

#45

I'm not sure about machine learning specifically, but I heard somewhere that a data scientist is someone who does statistics, on a Mac, in San Francisco.

After spending 80% of their time cleaning up garbage input data.

Re: What’s the difference between statistics and machine learning?

#46
post #39
post #29

Earlier quoted context omitted.

I think what you're saying is orthogonal to what I'm saying. Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem…

> machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved The NFL theorems essentially state that for every dataset on which an algorithm generalizes well, there is another on which it generalizes poorly. So, there are always implicit biases, even for a random forest model (eg: if you try to model Boolean functions with a…

> And that is precisely why I don’t see how ML algorithms are “more powerful” than statistics in any way, as per your claim.

Well I'm glad you don't see that, but I'm a bit confused why you think I said it because I didn't say it and don't believe it. I said that ML is "powerful," not "more powerful."

Re: What’s the difference between statistics and machine learning?

#47
post #11

That's easy - statisticians take pride in models that are understandable, while machine learning practitioners take pride in models that are not.

Understandable? Two words: statistical significance :P I've seen university employees who still don't understand what it is and can't explain it...and just about everyone who uses it gets it wrong...

Most scientists who use statistics are not statisticians.

Re: What’s the difference between statistics and machine learning?

#48
One difference is that in Machine Learning you must think of data structures and algorithms. i.e. the practical ways to compute a model. How to represent and transform data while building a model. I think this is given less emphasis in statistics. Standard models are often used and theory is built around these different models. For example aspects such as power calculations for a regression model.

Re: What’s the difference between statistics and machine learning?

#50
post #12

I can't speak for all of machine learning, but classic statistics has some probabilistic assumptions and uses those to prove some theorems about the results of how statistical methods manipulate data. E.g., in regression analysis, the assumptions are (i) there really is a linear mode with the variables to be used; (ii) typically the data will not fit the model exactly and instead there are errors; (iii) the errors ar…

To a rough approximation they differ in the theorems/results they care about. In statistics one would care about consistent estimate of parameters -- as sample size goes to infinity and the model is such and such here is an estimator that will converge (according to some interesting mode of convergence) to the true parameter generating the data.

In ML one wouldnt care much about recovering the parameters. The results/theorems of interest would be that with large enough samples the predictions and the new data will converge (according to some interesting mode of convergence). If this comes at a cost of doing poorly in terms of parameter recovery, ML wouldnt be bothered.

According to ML the cycloids and epicycloid based geocentric model of planetary motion would be perfectly acceptable.

Post reply on HN