Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are ML and Statistics Complementary? [pdf]

ics.uci.edu

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Are ML and Statistics Complementary? [pdf]

#21

Post by jupiter90000 »

p4wnc6 wrote: post #20
jupiter90000 wrote: post #9

Earlier quoted context omitted.

I hear what you're saying, but in terms of usefulness for business decisions, what leaders at a company would be satisfied with someone providing Vapnik's quote? Certainly machine learning and statistics have applications outside of business, but when it comes to realities in industry settings, very often an explanation of results is necessary, in addition to an explanation of why a particular machine learning approa…

The problem is not about how to take a scientific conclusion and make it suitably less scientific to serve as a business explanation. The problem is to replace inept employees who believe "business decisions" are not scientific questions, so that over time there is a convergence to using the scientific method, with legitimate statistical rigor, when making a so-called business decision. Generally speaking, the only p…

Edit: I wrote prior to this edit agreeing with alot of what you said. However, I mostly wanted to say that providing an explanation that people may be able to understand more easily doesn't necessarily make a scientific conclusion less scientific.

Re: Are ML and Statistics Complementary? [pdf]

#22

Post by sjg007 »

p4wnc6 wrote: post #8

What is commonly understood as 'statistics' is just a specialized subset of machine learning. Machine learning generalizes statistics. The correct complement to machine learning is cryptography -- trying to intentionally build things that are provably intractable to reverse engineer.

Machine learning does not generalize statistics; mathematics does.

Re: Are ML and Statistics Complementary? [pdf]

#23

Post by jupiter90000 »

_0w8t wrote: post #19
51109 wrote: post #16

Earlier quoted context omitted.

I don't think the paper asked for that. Relevant quote: > machine learning is more concerned with making predictions, even if the prediction can not be explained very well (a.k.a. “a black­box prediction”) So in your example: an algo may explain that a car slows down, before taking a turn, because else it would likely crash. It may even get to a threshold ("under these weather conditions, anything over 55Mph is unsaf…

I am referring to this fragment: > Thus, for many applications, in order to successfully interact with humans, machines will need to explain their reasoning, including some quantification of confidence, to humans. No doubtful there are cases when an explanation is easy. Often this is because we have a very solid model like physics of a car. In fact since we know the model, we do not need an explanation, we must deman…

Maybe you couldn't always expect an explanation, but having one would certainly be useful. Methods that could get the job done while also providing enough information to explain or at least shed some kind of light on why the solution was chosen would likely be preferable.

Re: Are ML and Statistics Complementary? [pdf]

#24

Post by murbard2 »

nextos wrote: post #6

I think they will eventually converge. Probabilistic programming is already a hint of this. The most general class of probability distributions is that of non-deterministic programs. ML is just a quick and dirty way to write these programs.

It's not just a way to write them, it's a way to do inference. Probabilistic programming is extremely powerful in terms of representation but inference is, in general, intractable. Yes, you can express all those ML models as probabilistic programs, but the sampler isn't going to perform nearly as well as the original algorithm.

Re: Are ML and Statistics Complementary? [pdf]

#25

Post by msellout »

p4wnc6 wrote: post #17
51109 wrote: post #13

Earlier quoted context omitted.

Working with both statisticians and pure machine learners on the same task, I did notice some tendencies, presuppositions and modus operandi that were different (beyond being a specialized subset). Like said in this position paper, machine learners like to throw computation and parameters at the problem, where statisticians are more careful and sober. As an analogy, a statistician will approach a cliff very carefully…

I have had the exact opposite experience with machine learning and statistics. In my experience, those who come from the 'statistics' side tend to use constructs, like null hypothesis significance testing, which are not consistent even from a theoretical point of view. And further, when they use them, they do awful things like p hacking, or using a direct comparison of t-stats as a model selection criterion, which ar…

You may have formed your generalization about statisticians from a biased sample. Or perhaps you're conflating statistics (ab)users for statisticians. There are far more people who have heard of a t-stat and r-squared than people I would call statistician.

Re: Are ML and Statistics Complementary? [pdf]

#26

Post by p4wnc6 »

msellout wrote: post #25
p4wnc6 wrote: post #17

Earlier quoted context omitted.

I have had the exact opposite experience with machine learning and statistics. In my experience, those who come from the 'statistics' side tend to use constructs, like null hypothesis significance testing, which are not consistent even from a theoretical point of view. And further, when they use them, they do awful things like p hacking, or using a direct comparison of t-stats as a model selection criterion, which ar…

You may have formed your generalization about statisticians from a biased sample. Or perhaps you're conflating statistics (ab)users for statisticians. There are far more people who have heard of a t-stat and r-squared than people I would call statistician.

I disagree. Most of the egregious stuff is in published statistics literature, particularly in econometrics, psychology, medicine, and biology, from researchers whose full-time job is to use statistics to solve applied problems ("domain statisticians" if you will).

Even if your definition of "statistician" only applied to Wasserman or Gelman types, I'd still say that the machine learning folks of the same level exhibit hugely more caution about the theoretical properties of their models (not a knock against Wasserman or Gelman, just a property of the rigor of e.g. PAC learning versus some ad hoc hierarchical model).

Re: Are ML and Statistics Complementary? [pdf]

#27

Post by p4wnc6 »

sjg007 wrote: post #22
p4wnc6 wrote: post #8

What is commonly understood as 'statistics' is just a specialized subset of machine learning. Machine learning generalizes statistics. The correct complement to machine learning is cryptography -- trying to intentionally build things that are provably intractable to reverse engineer.

Machine learning does not generalize statistics; mathematics does.

Machine learning is one subfield of mathematics that generalizes another, further subfield (statistics).

Re: Are ML and Statistics Complementary? [pdf]

#28

Post by p4wnc6 »

jupiter90000 wrote: post #21
p4wnc6 wrote: post #20

Earlier quoted context omitted.

The problem is not about how to take a scientific conclusion and make it suitably less scientific to serve as a business explanation. The problem is to replace inept employees who believe "business decisions" are not scientific questions, so that over time there is a convergence to using the scientific method, with legitimate statistical rigor, when making a so-called business decision. Generally speaking, the only p…

Edit: I wrote prior to this edit agreeing with alot of what you said. However, I mostly wanted to say that providing an explanation that people may be able to understand more easily doesn't necessarily make a scientific conclusion less scientific.

True. It's not necessary. But most times in real life when this happens, the scientific rigor is the first thing to go. This can manifest itself in a lot of ways. One way is an explicit mandate to find that a pre-determined conclusion is supported, even if the data don't support it. Another way is to place a greater emphasis on speed of delivery than on accuracy, and to avoid quantifying the true trade-off between the two by invoking the magic buzzword "business concerns" or "bottom line" or whatever.

Yet another way is that data analytics platforms are built from the ground up with hard-wired priorty given to scaling out the ability to test multiple hypotheses without any attempt to correct the significance metrics for the multiplicity of testing (or, even subtler, for subject researcher degrees of freedom that further affect the multiplicity of testing). Often, the business stakeholders who are demanding such an "analytics" system aren't even aware of the statistical fallacies they are inexorably baking right into the platform itself (one might call this the "Hadoop disease", though it's not stricly the fault of Hadoop or Hadoop-like tools).

At any rate, I would say in the current climate of "analytics" in business environments, to a good first approximation, one can assume that "make it easy to understand" is exactly equivalent to "throw out any and all difficult yet rigorous science until the thing is cheap and easy, and then just use that."

Re: Are ML and Statistics Complementary? [pdf]

#29

Post by jupiter90000 »

p4wnc6 wrote: post #28
jupiter90000 wrote: post #21

Earlier quoted context omitted.

Edit: I wrote prior to this edit agreeing with alot of what you said. However, I mostly wanted to say that providing an explanation that people may be able to understand more easily doesn't necessarily make a scientific conclusion less scientific.

True. It's not necessary. But most times in real life when this happens, the scientific rigor is the first thing to go. This can manifest itself in a lot of ways. One way is an explicit mandate to find that a pre-determined conclusion is supported, even if the data don't support it. Another way is to place a greater emphasis on speed of delivery than on accuracy, and to avoid quantifying the true trade-off between th…

Thanks for your perspective. It is disheartening to see what you're talking about happen. I've definitely witnessed this first-hand and it was frustrating to me to essentially see lots of rigorous scientific work be dismissed because of someone's ego or because someone making the decisions couldn't wrap their brain around it.

What I've seen around this is analytics professionals hired under the pretense that their skills to produce accurate scientific conclusions will be used for the good of a business, yet having their conclusions and efforts dismissed for no good reason other than decision makers 'didn't get it' or otherwise just refused to heed the results. So why did they hire experts in the first place, then? To lend the company credibility that it doesn't really deserve? I'm sure lots of the reason for this type of thing is politically motivated, as you previously mentioned.

Re: Are ML and Statistics Complementary? [pdf]

#30

Post by 51109 »

p4wnc6 wrote: post #17
51109 wrote: post #13

Earlier quoted context omitted.

Working with both statisticians and pure machine learners on the same task, I did notice some tendencies, presuppositions and modus operandi that were different (beyond being a specialized subset). Like said in this position paper, machine learners like to throw computation and parameters at the problem, where statisticians are more careful and sober. As an analogy, a statistician will approach a cliff very carefully…

I have had the exact opposite experience with machine learning and statistics. In my experience, those who come from the 'statistics' side tend to use constructs, like null hypothesis significance testing, which are not consistent even from a theoretical point of view. And further, when they use them, they do awful things like p hacking, or using a direct comparison of t-stats as a model selection criterion, which ar…

Thanks for the links to interesting papers. I really liked the Breiman paper. I did not try to qualify either machine learners and statisticians as bad or good, just pointing out a difference in their approaches to problems.

I do not know enough about statistics to make a (negative) quality statement about it. I know a bit more about machine learning though, and there I also see things like: Picking the most favorable cross-validation evaluation metric, comparing to "state-of-the-art" while ignoring the real SotA, generating your own data sets instead of using real-life data, improving performance by "reverse engineering" the data sets, reporting only on problems where your algo works, and other such tricks. I believe you when you say much the same is happening for statisticians.

Maybe it was my choice of words (careful, sober). I think its fair to say that (especially applied) machine learners care more about the result, and less about how they got to that result. Cowboys, in the most positive sense of the word. I retraced where I got the cliff analogy. It's from Caruana in his video "Intelligible Machine Learning Models for Health Care" https://vimeo.com/125940125 @37:30.

"We are going too far. I think that our models are a little more complicated and higher variance than they should be. And what we really want to do is to be somewhere in the middle. We want this guy to stop and we want that statistician to get there, together we will find an optimal point, but we are not there yet."

Post reply on HN