What’s the difference between statistics and machine learning?

statisticians take pride in models that are understandable, while machine learning practitioners take pride in models that have high accuracy.

I personally don't make the distinction between statistics and machine learning that your question seems predicated on.

Also I rarely find it useful to distinguish between theory and practice; their interplay is already profound and will only increase as the systems and problems we consider grow more complex.

Think of the engineering problem of building a bridge. There's a whole food chain of ideas from physics through civil engineering that allow one to design bridges, build them, give guarantees that they won't fall down under certain conditions, tune them to specific settings, etc, etc. I suspect that there are few people involved in this chain who don't make use of "theoretical concepts" and "engineering know-how". It took decades (centuries really) for all of this to develop.

Similarly, Maxwell's equations provide the theory behind electrical engineering, but ideas like impedance matching came into focus as engineers started to learn how to build pipelines and circuits. Those ideas are both theoretical and practical.

We have a similar challenge---how do we take core inferential ideas and turn them into engineering systems that can work under whatever requirements that one has in mind (time, accuracy, cost, etc), that reflect assumptions that are appropriate for the domain, that are clear on what inferences and what decisions are to be made (does one want causes, predictions, variable selection, model selection, ranking, A/B tests, etc, etc), can allow interactions with humans (input of expert knowledge, visualization, personalization, privacy, ethical issues, etc, etc), that scale, that are easy to use and are robust. Indeed, with all due respect to bridge builders (and rocket builders, etc), but I think that we have a domain here that is more complex than any ever confronted in human society.

I don't know what to call the overall field that I have in mind here (it's fine to use "data science" as a placeholder), but the main point is that most people who I know who were trained in statistics or in machine learning implicitly understood themselves as working in this overall field; they don't say "I'm not interested in principles having to do with randomization in data collection, or with how to merge data, or with uncertainty in my predictions, or with evaluating models, or with visualization". Yes, they work on subsets of the overall problem, but they're certainly aware of the overall problem. Different collections of people (your "communities") often tend to have different application domains in mind and that makes some of the details of their current work look superficially different, but there's no actual underlying intellectual distinction, and many of the seeming distinctions are historical accidents.

I also must take issue with your phrase "methods more squarely in the realm of machine learning". I have no idea what this means, or could possibly mean. Throughout the eighties and nineties, it was striking how many times people working within the "ML community" realized that their ideas had had a lengthy pre-history in statistics. Decision trees, nearest neighbor, logistic regression, kernels, PCA, canonical correlation, graphical models, K means and discriminant analysis come to mind, and also many general methodological principles (e.g., method of moments, which is having a mini-renaissance, Bayesian inference methods of all kinds, M estimation, bootstrap, cross-validation, EM, ROC, and of course stochastic gradient descent, whose pre-history goes back to the 50s and beyond), and many many theoretical tools (large deviations, concentrations, empirical processes, Bernstein-von Mises, U statistics, etc). Of course, the "statistics community" was also not ever that well defined, and while ideas such as Kalman filters, HMMs and factor analysis originated outside of the "statistics community" narrowly defined, there were absorbed within statistics because they're clearly about inference. Similarly, layered neural networks can and should be viewed as nonparametric function estimators, objects to be analyzed statistically.

In general, "statistics" refers in part to an analysis style---a statistician is happy to analyze the performance of any system, e.g., a logic-based system, if it takes in data that can be considered random and outputs decisions that can be considered uncertain. A "statistical method" doesn't have to have any probabilities in it per se. (Consider computing the median).

When Leo Breiman developed random forests, was he being a statistician or a machine learner? When my colleagues and I developed latent Dirichlet allocation, were we being statisticians or machine learners? Are the SVM and boosting machine learning while logistic regression is statistics, even though they're solving essentially the same optimization problems up to slightly different shapes in a loss function? Why does anyone think that these are meaningful distinctions?

I don't think that the "ML community" has developed many new inferential principles---or many new optimization principles---but I do think that the community has been exceedingly creative at taking existing ideas across many fields, and mixing and matching them to solve problems in emerging problem domains, and I think that the community has excelled at making creative use of new computing architectures. I would view all of this as the proto emergence of an engineering counterpart to the more purely theoretical investigations that have classically taken place within statistics and optimization.

But one shouldn't definitely not equate statistics or optimization with theory and machine learning with applications. The "statistics community" has also been very applied, it's just that for historical reasons their collaborations have tended to focus on science, medicine and policy rather than engineering. The emergence of the "ML community" has (inter alia) helped to enlargen the scope of "applied statistical inference". It has begun to break down some barriers between engineering thinking (e.g., computer systems thinking) and inferential thinking. And of course it has engendered new theoretical questions.

I could go on (and on), but I'll stop there for now...

And since we cannot define exactly what this value is (accuracy, RMSE, bias or a specific confusion matrix), we have to make a more abstract definition.

We are now left with a question. Is this abstract definition quantifiable? If so, we can still exclude the statistician w.r.t. understandably. However, if we allow qualitative value as well, the traditional statistician is back at the table.

Now, a further refinement needs to be made. Since fairness, accountability and transparency in Data Science are in the limelight, we can make a point that our ability to understand is a key metric for Machine Learning models as well. It is interesting to see the tendency to associate words or phrases to key neurons in embeddings. For example, in embeddings of faces, we can associate gender, ethnicitity, fatness and gaze with some of the embedded neurons.