Statisticians want to abandon science’s standard measure of ‘significance’

It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis.

This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.

The current approach also ties the experimental method to the result, requiring potentially fallible oversight of the experimental design. The hypothesis-testing aggregator reduces this reliance, which is perhaps more honest about our ability to design good experiments, would allow for a greater diversity of approaches, and could also be more efficient when evaluating results.

The most interesting aspect of this is if you could somehow create relationships between standardized hypotheses found on the hypothesis menu such that you could infer other hypotheses in a more systematic way.

That's called picking a prior. Usually, pick the prior that maximizes the entropy on the given interval (zero prior knowledge).

For those reading and confused. Computing: p(H|x) = p(x|H)p(H)/p(x)

requires p(H) which the parent is suggesting is impossible to grasp.

If we want to know the probability that a coin is biased, we can assign probabilities to each hypothesis. For some people, p(bias=1/2)=1 (Dirac distribution), others might argue that p(bias=x)=1 for x in [0,1] (uniform distribution). Others might argue its some Beta function centered around 1/2. I believe what the parent is suggesting is that choosing which original belief we have in the system is a matter of philosophy, not computation.

P hacking isn't always done on purpose. It's misunderstanding. That's probably why it still passes peer review. There's also tons of incentives that encourage this behavior. These are the problems. An arbitrary mark to meet makes people weak at statistics not understand their data as well. I'd rather more accurate data than more scientists. More workers doing poor work isn't useful.

Also, you can Bayes hack. Bayes helps, but it doesn't address the underlying issues.

The physics community is pretty good about dealing with both of these shortcomings. You often need a lower p-value of around p=0.003 (3 sigma) or lower to say something was significant, and a few more sigmas to claim an actual discovery. And in addition to this, you're expected to include a trials factor to correct for multiple hypotheses. It's not perfect, but it makes claims of statistical significance more meaningful.

No we can't. There are 2 (related) reasons for this. The first is that to say "after experiment X, we can conclude thing Y is happening with probability ..." you need to know something about how often thing Y happens on its own. This is also known as a prior. The other reason is that in this specific case, the prior for Y (that the coin is weighted) can be concisely summarized as P(Y)=0, because it's physically impossible to construct a coin that's biased to one side or the other (independent of what side it starts on). Some people think the coin flipping examples are bad because biased coins are impossible, but they're actually good for exactly this reason. We have a really good prior (from math/physics/geometry) that biased coins can't exist, so you're going to have to not only flip a lot of them to convince me otherwise, but also give me an explanation of how that's possible (i.e justify your prior)