Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians want to abandon science’s standard measure of ‘significance’

sciencenews.org

Post reply on HN
Print view View original

81–90 of 142 posts

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#81

Post by buboard »

jerf wrote: post #59
buboard wrote: post #17

Earlier quoted context omitted.

> "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false? The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.

You've mangled my sentence in your quote. The rest matters. It may not be a masterpiece of the English language, but I already said the first is not directly possible and I wrote the second as a description of p-values quite on purpose. "Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used." That's begging the question. The entire topic…

ah yes i agree with those. As you said, the selection of hypothesis is the actual problem, and statistical power doesnt solve that one either.

There are some scientists thinking over this though. This is an idea from a neuroscience lab: https://www.researchmaps.org/

The idea was to create causal directed graphs for biology from the literature , which would be used identify what experiments are missing and thus inform future science.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#82

Post by buboard »

merpnderp wrote: post #52
buboard wrote: post #20

Earlier quoted context omitted.

Scientists have a duty to communicate what is true and what is not though. Passing that judgement duty to the general public is irresponsible and unwise (it's easy to claim anything with flawed / misleading statistics). Ideally someone would have come up with a framework that is better than statistic tests to justify levels of truth, but so far we dont have one

Richard Feynman said anything you can't prove yourself must be taken on pure faith and most of the public must take everything on pure faith as somehow they've left school without the faintest idea how basic stats work.

Feynman was talking to scientists. I do think that science has to provide answers, not just data. The public interprets every datum in a myriad of ways, and does not have a mechanism to establish consensus, hence you would end up with constant crisis.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#83

Post by buboard »

bigfudge wrote: post #41
buboard wrote: post #29

Earlier quoted context omitted.

There are very few principles in science. Statistics tests seem to be something where scientists unanimouslu agree. Another is models, and applying occam's razor to choose the one with less parameters. Perhaps one could also justify choosing a hypothesis on the basis that it is simpler.

They certainly don’t agree. There has been 30+ years of objection to p < .05. The fact that many scientists are statistically illiterate doesn’t constitute consensus.

Scientists (should) never really "agree", they just concede temporarily preparing their next attack. statistic tests are the de facto standard that everyone concedes to, and has not been replaced.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#84

Post by WhompingWindows »

TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

P-values are much more problematic at the alpha=.05 threshold than the alpha=.005 or alpha=.01 thresholds.

In my work as a data scientist, 99% of the time this is not even relevant. We have huge sample sizes and most of the p-values we see are The question then: does a mean of 20.02 being significantly different from a mean of 20.03 over 10 million patients, is that actually important to us?

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#85

Post by ghkbrew »

btilly wrote: post #77
TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

I disbelieve. The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs. Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly…

> But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences

That's not how statistics works. Frequentist p-values and Bayesian inferences are both entirely dependent on the mathematical model in question. "Their plans if something else had happened" are a key part of the model. What you're describing at 2 entirely different experiments:

1) A couple has children until they have 1 of each gender. they have 7 children. How likely is a result this extreme (or greater)?

2) A couple has 7 children. They have a gender ratio of 6:1. How likely is a result this extreme (or greater)?

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#86

Post by xenocyon »

kerkeslager wrote: post #16

If you're looking for a replacement you don't understand the problem. The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis . Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up he…

This is correct, and also illustrates why merely replacing frequentist methods with Bayesian methods is not sufficient to fix the scientific reproducibility crisis.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#87

Post by xenocyon »

edmundsauto wrote: post #78
arafa wrote: post #65

I am pretty surprised to see no discussion of statistical power in the article and very little mention in the comments here. To me, having more statistical power solves many of the issues mentioned in the article. Many of the rest can be handled with use of Bayesian priors, context-specific p-values thresholds (.01, .1, etc.), and replication. There's decent working guidelines for statistical power, but a lot of the…

Can you recommend any texts, videos, or websites to learn more about statistical power? The term is quite overloaded in google.

A quick 101: https://www.statisticsdonewrong.com/power.html

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#88

Post by srpguy »

btilly wrote: post #77
TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

I disbelieve. The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs. Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly…

"You can give any number of lectures on what it actually means - I guarantee that it will be misunderstood."

So true. That said, here's a great interactive explaining the p-value: https://www.jwilber.me/permutationtest/

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#89

Post by rjf72 »

TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

I agree with you, but I think there is an even more fundamental issue. P-values really don't tell you anything. They can refute a bad hypothesis but they cannot validate a good one. This [1] is such an awesome site. It lists a large number of spurious correlations. It's quite remarkable how strong many of them are. For instance there is a 99.79% correlation between suicides by hanging and US spending on science/space/technology.

The big problem is that since we've already long since entered the world of big data you can now find spurious correlations, similarly strong, but constrained to variables that are plausibly connected. For instance what if this correlation was between spending on science/space/technology and an increase in the number of people pursuing STEM fields? People would immediately just accept it without question, even though it should in theory be held to the same scientific rigor and critique as one that challenges our biases. Science should not be a glorified exercise in confirmation bias. Spurious correlations don't mean you have a biased sample or that such correlations don't exist - it simply means that correlations have no direct relationship.

538 has a great little interactive p-hacking game you can play, with real data. [2] Ultimately any single datum that tries to legitimize a piece of research is going to be gamed. Science should be judged as it was in its 'glory days' - by the logic, predictability, and falsifiability of the ideas presented. Anything that steps outside these bounds should be treated with the most extreme of prejudice. It may be legitimate but the great burden of proof is on the presenter, and that doesn't come down to p-hacking up a 0.00001 correlation and calling it causal.

[1] - http://www.tylervigen.com/spurious-correlations

[2] - https://projects.fivethirtyeight.com/p-hacking/

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#90

Post by arafa »

xenocyon wrote: post #87
edmundsauto wrote: post #78

Earlier quoted context omitted.

Can you recommend any texts, videos, or websites to learn more about statistical power? The term is quite overloaded in google.

A quick 101: https://www.statisticsdonewrong.com/power.html

See also this, for more specifics on the math involved: https://effectsizefaq.com/2010/05/31/what-is-statistical-pow...

The basic idea is: Get more data, especially if you're measuring something subtle.

Post reply on HN