Machine Learning Is the New Statistics

Deep Learning wasn't even possible until recently though, data and compute power have made it possible as opposed to just proportionally better [1]. There have also been a lot of breakthroughs in Reinforcement Learning riding on the wave of Deep Learning, and both of those (DL and RL) are more than applied statistics.

1. I think of this as a 0 to 1 innovation versus a 1 to n innovation if you're familiar with Peter Thiel's terminology on that.

Well, each random variable has a distribution. And there are a few distributions that are common so are taught. Then, presto, bingo, too many students conclude that an important first step is to find a distribution. However, commonly in practice, with just samples and without more in mathematical assumptions, finding a distribution is from not very promising to hopeless. Hopeless? Yes, consider a random variable that takes values in 50 dimensional Euclidean space.

But there is a lot of statistics that is distribution-free, where we make no assumptions on probability distributions. E.g., I published such a paper in Information Sciences. In addition with some meager assumptions, say, the random variables have expectations, the squares of the random variables have finite expectations, etc., can do more.

For model fitting, if can assume that the data has Gaussian distribution, is homoscedastic, have independent and identically distributed (i.i.d.), etc., then can get some more results, e.g., know that some of the results of the computations have Gaussian or F distribution, etc. Then can do a lot of classic hypothesis tests, confidence intervals, etc.

But, with just meager assumptions, commonly can still proceed and know that are still making a best L^2 approximation. Then can drag out the classic result that a sequence of (such) random variables that are Cauchy convergent in L^2 do converge in L^2, that L^2 is complete (i.e., a Hilbert space), and that some sub-sequence converges almost surely. That's a lot -- might be able to take that to the bank. And made no more than meager, general assumptions about distributions.

Really, often we get some of the well known distributions from some theorems, not analysis of empirical data. E.g., get a Gaussian assumption from the central limit theorem. Can get an exponential distribution from the Poisson process (e.g., E. Cinlar's text), and can get that from the very general, even astounding, renewal theorem (e.g., W. Feller's second volume).

In fact the birth of statistical learning theory was Vapnik's (rather unintuitive) insight that although the theoretically optimal strategy in a classification task is the class conditional distribution, estimating the class conditional distribution from the data is not a promising way to do it. Estimating distributions is freaking hard, you may not (ever) have data to estimate that distribution, however you can short circuit the process and solve the classification problem directly (without explicitly modeling the class conditional distribution).

That said, machine learning has models that do not rely on a stochastic sampling process. The sampling process can be adversarial too. In such situations the algorithms guarantee that you wouldn't be too far off (even in finite time) from the strategy that would be optimal in hind-sight. These are non-asymptotic guarantees. I am of course sweeping large swathes of theory under the carpet, because this is hardly the forum for that.

The beauty of CLT notwithstanding, what is surprising is how rarely the Gaussian assumption holds in practice. Many distributions are too heavy tailed to have a finite variance, and in such scenarios CLT does not yield you a Gaussian but yields a stable distribution (Gaussian is just one, in fact the only one in the stable family with a finite distribution. Gaussian distribution was not discovered by Gauss and neither is it that normal). To make a tongue in cheek claim, the prince of mathematicians sort of got away with it.

Machine learning theory was never far from statistics and probability. How can it, when its traditional theoretical bedrock (PAC) is explicitly a statement about convergence in probability with all its epsilon and delta in your face. Where the two differ are in the amount of focus on prediction (as opposed to parameter estimation), familiarity with optimization and algorithms, and lack of fascination for test of hypothesis, Gaussian distribution and asymptotic normality.

Mind, huge data sets and huge amounts of computing power also make the use of Bayesian methods feasible, which don't have the NHST problem that anything is significant with enough data-points.