Statisticians want to abandon science’s standard measure of ‘significance’

> It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis.

You're absolutely right, but journals don't see it that way. Journals want to make money, and the articles which make the most money are the ones that prove an alternative hypothesis. This is one of the ways where for-profit publishing is harmful for science.

> This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.

"Standardization of hypotheses" gets a bit tricky and I suspect the standardization process would be stifling. Part of the goal of replication is to improve on methodology, and part of that is finding ways in which your hypothesis wasn't clearly defined, or doesn't contribute to a larger theory. There needs to be some flexibility in which hypotheses scientists pursue.

A more organic way might be for journals to categorize articles as testing new hypotheses or attempting to reproduce old results, and strive for some ratio between the two (1:4 or somesuch).

I have a rule of thumb: Multiply p by 10. In other words, p = 0.05 is as good as a coin toss. This is consistent with the so called replication crisis.

In my view, understanding p-values won't help. Feeding data into a formula that produces a p-value doesn't account for things that can go wrong with experiments: Uncontrolled experimental conditions, biased sampling, hidden correlations, and simply non-ergodic systems. It is possible that some systems can't be controlled to the point where it's reasonable to start looking for real effects.

I suspect that if the results were any good, the precise interpretation of the p-value wouldn't matter. Physics has lived without an agreed-upon interpretation of quantum theory for a century.

We already expect data to sometimes indicate significant evidence against the null hypothesis when the null hypothesis is actually correct. That's just the nature of the way we structure our statistics practices. But for each additional set of variables you have and could test for relationships, you increase your chances of finding a strong relationship that appears to be significant due to random chance, but was in fact not. (You may notice that this is just a different flavor of the issues that arise from the way researchers use significance tests).

The reason you need to pick your hypothesis ahead of time is because otherwise you may use intuition or just your eyeballs to find relationships in the data that are only there due to random chance. This will supposedly increase the probability of detecting spurious relationships and erroneously rejecting the null.

Thanks for sharing! (writes the girl who barely passed pre-calc)

Thanks, although I must admit I'm having some trouble following your logic.

IIUC, you're pointing out that when examining the empirical data gathered during an experiment, it's often possible to find some identifiable subset of the data that are consistent with a refined version of the original hypothesis. E.g., maybe jellybeans in general don't correlate with acne, but green ones do.

Assuming that the experiment is part of a larger effort to build or refine some model, I don't see the problem.

Suppose that someone refrained from p-hacking during the first run of that experiment. IIUC, they'd look at the experimental results, and wonder if they had missed some "X" factor. So they might conject that jellybean color was relevant, and rerun the experiment with the hypothesis, "consumption of jellybeans, but only of a particular color, correlates with acne." And (assuming sample sizes were big enough), the data gathered during that second experiment would likely confirm that green jellybeans correlate with acne.

But what's the point of having run that second experiment, when they could have just reached the same conclusion by testing additional hypotheses from the first experiment's data?

It seems like regardless of whether you just ran one experiment and did "p-hacking", or instead ran a follow-on experiment, you end up with the same refinements to the model you're working on.