Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article).

Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether.

Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas more sophisticated statistical techniques would be difficult without any background in linear algebra or calculus.

Unfortunately, this enables researchers in all sorts of fields to make incredibly spurious statistical analyses that look convincing, because "all the calculations are right", even though they're using completely the wrong tool.

Andrew Gelman, quoted in the article, feels very strongly that F-tests are always unnecessary[0]. I'd go as far as to extend that logic to the Student's t-test and any other related test as well.

You can get into all sorts of confusing "paradoxes" with p-values. One of my favorites:

Alice wants to figure out the average height of population. Her null hypothesis is 65 inches. She conducts a simple random sample, performs a t-test, and determines that the sample mean is 70 inches, with a p-value of .01.

In an alternate universe, Bob does the same thing, with the same null hypothesis (65 inches). He determines that the sample mean is 90 inches, with a p-value of .000001.

Some questions:

A) Does Bob's experiment provide stronger evidence for rejecting the null hypothesis than Alice's does?

B) In Bob's universe, is the true population mean higher than it is in Alice's universe?

By pure hypothesis testing alone, the correct answer to both questions is "no", even though the intuitive answer to both questions is "yes"[1].

[0] http://andrewgelman.com/2009/05/18/noooooooooooooo/

[1] Part of the problem is that we do expect that, in Bob's universe, the true population mean is highly likely to be higher, and this is supported by the data. Trouble is, the reason we expect that is not formally related to hypothesis testing and t-tests/p-values.

Wow, I remember having these reservations about p-values when I took classes in stats but whenever I brought them up a prof. would wave their hands and be dismissive. They gave me a degree in political science, but I felt that political science was an oxymoron and it left me with no respect for the field.

I agree as well!

Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs.

p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to rethink things. This is not a valid statistical procedure, but it IS how we naively think. And the substitution of a complex question for a simpler one is exactly how our brains are set up to handle complex questions about our environment. (I'm currently working through Thinking Fast and Slow which has a lot to say about this.)

Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different.

Both approaches have a fundamental mistake. The mistake is that we are taking a data set and asking what it TELLS us about the world. When in probability theory the real role of data is how to UPDATE our views about the world.

This is why I have come to believe that for simple A/B testing, thinking about p-values is a mistake. The only three pieces of information that you need is how much data you are willing to collect, how much you have collected, and how big the performance difference is. Stop either when you have hit the maximum amount of data you're willing to throw at the test, or when the difference exceeds the square root of that maximum amount. This is about as good as any simple rule can do.

If you try to be clever with p-values you will generally wind up putting yourself at saving yourself some effort in return for a small risk per test of making very bad mistakes. Accepting small risk per test over many tests for a long time puts you at high odds of eventually making a catastrophic mistake. This is a very bad tradeoff.

I've personally seen a bad A/B test with a low p-value rolled out that produced a 15% loss in business for a company whose revenues were in the tens of millions annually. It. Was. Not. Pretty. (The problem was eventually found and fixed..a year later and after considerable turnover among the executive team.)

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it?

I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

> 5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so.

On the other hand, let's say you do a study with 20 people with cancer. You give 10 of them a drug, the other 10 a placebo. All 10 with the drug survive; all 10 with the placebo die. Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be.

This is all to say that the misuse of P-values does not just come from the threshold.