Statisticians want to abandon science’s standard measure of ‘significance’

The problem with p-values is that the shape of the resulting cutoff just isn't what we're looking for. We want something along the lines of "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?" is only an approximation at the best of times, and at worst, downright misleading. That's true even before we consider some of the other issues that my English kind of elides over, but the math contains; one of my problems with significance testing as it is commonly done is that there are actually ranges of hypotheses, and it really overprivileges "the" "null" hypothesis; I could write a decently-sized HN post just criticizing those two quoted words.

There is no rigid process that can produce the answer we really want, but I think we can provide a selection of better default tools. An example of one that has already been deployed to some extent is "power analysis", which is not a direct answer but lets people crafting studies analyze how big their studies will have to be before running them. We can build more tools like that.

The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis.

Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up heads with P=.5. If you flip the coin twice, the null hypothesis is that both flips come up heads with P=.25. The probability of all coins coming up heads is P=0.125 for 3 flips, P=0.0625 for 4 flips, and P=.03125 for 5 flips. So if we flip a coin 5 times, and get heads all 5 times, we can conclude that the coin is weighted in some way with P=0.03125.

Let's say all the major journals of coin flipping only publish results with the high significance of PNow somewhere down the line, Robert flips a coin 5 times and gets 5 heads. This result has P=.03125, which meets the requirement of PNature and Science magazines do front page pieces with headlines "Quarters Found Heavy-headed" and "Washington Shows His Face" respectively. A casino hires Robert as a consultant for the design of their coin-flipping games. His quarter-flipping study is cited in the abstracts of two dime-flipping studies and a half-dollar flipping study. During the trial of a murderer who placed quarters tails-side up on his victims, Robert is called as an expert witness to say that the coins were placed there, not flipped there.

A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125.

Now, if you survey the AJCFS and all the other academic journals on coin flipping, you'd conclude that quarters are significantly weighted towards heads. But in fact, the null hypothesis is true: quarters are pretty evenly weighted. Robert's low-P result is exactly what you'd expect to happen eventually if you have enough people perform the 5-quarter-flip experiment--in fact, if a lot of people are studying coin flips, the P of getting a low-P result approaches P=1. But because the AJCFS has a P=.05 requirement, they've created a bias against the null hypothesis, which deceives the public into thinking that flipped quarters are more likely to come up heads than tails.

This is likely the reason why so many fields, most notably psychology[2], are having a replication crisis[3] and a similar effect can be used in P-hacking[4] to bolster results that are essentially fake.

Unlike coin-flipping, fields with replication crises like psychology and medicine have real affects on real people's lives. It's irresponsible and unethical for journals to publish with a bias against the null hypothesis, and adjusting the P-value requirements to another significance requirement, even a less arbitrary one, doesn't fix the issue.

The solution, I think, is for journals to commit to publish studies before the study has been performed, based on the methodology, previous studies on the subject, and qualifications of the researcher. This would mean that many, many studies would be published with null results, and that would be a good thing.

[1] There are 32 possible outcomes for flipping a coin 5 times. If we group them by how many heads and tails, we can calculate a probability for each outcome:

    HHHHH 1 result of 5 heads       ->      P  = 1/32 = .03125

    HHHHT
    HHHTH
    HHTHH 5 results of 4 heads, 1 tails  -> P  = 5/32 = .15625
    HTHHH
    THHHH

    HHHTT
    HHTHT
    HHTTH
    HTHHT
    HTHTH
    HTTHH 10 results of 3 heads, 2 tails -> P = 10/32 = .3125
    THHHT
    THHTH
    THTHH
    TTHHH

    HHTTT
    HTHTT
    HTTHT
    HTTTH
    THHTT 10 results of 2 heads, 3 tails -> P = 10/32 = .3125
    THTHT
    THTTH
    TTHHT
    TTHTH
    TTTHH

    HTTTT
    THTTT
    TTHTT 5 results of 1 heads, 4 tails  -> P  = 5/32 = .15625
    TTTHT
    TTTTH

    TTTTT 1 result of 5 tails       ->      P  = 1/32 = .03125

[3] https://en.wikipedia.org/wiki/Replication_crisis

[4] https://journals.plos.org/plosbiology/article?id=10.1371/jou...

EDIT: Also see roenxi's excellent post on how "significant" means different things in statistics and colloquial English: https://news.ycombinator.com/item?id=20895893

It's not the statistical analysis that's the problem, it's the bad "science" and irresponsible journalism.