P < 0.05 Considered Harmful

1. Effect sizes.

Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one has been tried on 10000 people, and it gave an average benefit of 0.02 units with a standard deviation of 1 unit, on some 5-point scale. So the standard deviation of the average over 10k people is about 0.01 units, the average benefit is about 2 sigma, and p is about 0.05. Very nice.

The other drug has only been tested on 100 people. It gave an average benefit of 0.1 unit with a standard deviation of 0.5 units. Standard deviation of average is about 0.05, average benefit is about 2 sigma, p is again about 0.05.

Are these two interventions equally promising? Heck no. The first one almost certainly does very little on average, and does substantially more harm than good about half the time. The second one is probably about 5x better on average, and seems to be less likely to harm you. It's more uncertain because the sample size is smaller, and for sure we should do a study with more patients to nail it down better, but I would definitely prefer the second drug.

(With those very large standard deviations, if the second drug didn't help me I would want to try the first one, in case I'm one of the lucky people it gives > 1 unit of benefit to. But it might well be > 1 unit of harm instead.)

Looking only at p-values means only caring about effect size in so far as it affects how confident you are that there's any effect at all. (Or, e.g., any improvement on the previous best.) But usually you do, in fact, care about the effect size too.

Here's another way to think about this. When computing a p-value, you are asking "if the null hypothesis is true, how likely are results like the ones we actually got?". That's a reasonable question. But you will notice that it makes no reference at all to any not-null hypothesis. p not in fact quite what it means) but usually you also care how wrong it is, and the p-value won't tell you that.

2. Prior probability.

The parenthetical remark in the last paragraph indicates another way in which the p-value is fundamentally the Wrong Thing. Suppose your null hypothesis is "people cannot psychically foretell the future by looking at tea leaves". If you test this and get a p=0.05 "positive" result, then indeed you should probably think it a little more likely than you previously did that this sort of clairvoyance is possible. But if you are a reasonable person, your previous opinion was a much-much-less-than-5% chance that tasseomancy actually works[1], and when someone gets a 1-in-20 positive result you should be thinking "oh, they got lucky", not "oh, it seems tasseomancy works after all".

[1] By psychic powers, anyway. Some people might be good at predicting the future and just pretend to be doing it by reading tea leaves, or imagine that that's how they're doing it.

3. Model errors.

And, of course, if someone purporting to read the future in their tea leaves does really well -- maybe they get p=0.0000001 -- this still doesn't oblige a reasonable person to start believing in tasseomancy. That p-value comes from assuming a particular model of what's going on and, again, the test makes no reference to any specific alternative hypothesis. If you see p=0.0000001 then you can be pretty confident that the null hypothesis's model is wrong, but it could be wrong in lots of ways. For instance, maybe the test subject cheated; maybe that probability comes from assuming a normal distribution but the actual distribution is much heavier-tailed; maybe you're measuring something and your measurement process is biased, and your model assumes all the errors are independent; maybe there's a way for the test subject to get good results that doesn't require either cheating or psychic powers.

None of these things is helped much by replacing p=0.05 with p=0.001 or p=0.25. They're fundamental problems with the whole idea that p-values are what we should care about in the first place.

(I am not claiming that p-values are worthless. It is sometimes useful to know that your test got results that are unlikely-to-such-and-such-a-degree to be the result of such-and-such a particular sort of random chance. Just so long as you are capable of distinguishing that from "X is effective" or "Y is good" or "Z is real", which it seems many people are not.)

There's an argument for competitive games to use true randomness to eliminate any possibility of abuse, but I'm not aware of specific examples.