Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians want to abandon science’s standard measure of ‘significance’

sciencenews.org

Post reply on HN
Print view View original

41–50 of 142 posts

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#41

Post by bigfudge »

buboard wrote: post #29
mkl wrote: post #24

Earlier quoted context omitted.

When you've got empirically gathered data, pretty much every type of analysis you can do with it comes under the umbrella of "statistics". I think any better truth measuring framework can only be statistical.

There are very few principles in science. Statistics tests seem to be something where scientists unanimouslu agree. Another is models, and applying occam's razor to choose the one with less parameters. Perhaps one could also justify choosing a hypothesis on the basis that it is simpler.

They certainly don’t agree. There has been 30+ years of objection to p < .05. The fact that many scientists are statistically illiterate doesn’t constitute consensus.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#42

Post by blululu »

IMO the expression ‘statistical significance’ is a big part of the problem. Popular reporting of research translates a tiny but discriminable effect size into a SIGNIFICANT effect. Changing the nomenclature to ‘statistically discriminable’ would go a long way to improving popular understanding.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#43

Post by roenxi »

TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

We use hard cutoffs for a bunch of things, they aren't perfect but they are fine.

The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English.

It is not only possible, but likely that people will be producing results that are insignificant but statistically significant. I mean seriously, if I tell my boss that the results are statistically significant, how do I expect him to understand that the results might reasonably be insignificant? How do I expect anyone non-technical to take that sentence seriously? You lose all credibility pretty quickly when people start saying that the result is significant except it doesn't matter.

This is even more stupid than calling complex numbers "imaginary" and "complex". Those names are just arbitrary. Statistical significance is overloading words that are likely to come up with meanings other than what they are expected to mean.

Anyway, this is the problem that things like this article are going to. People are treating significance values as though they are significant. They aren't significant, they are statistically significant. It means something different to significance.

For example; if I run an experiment 20 times and get a p value consistent with a 5% chance in one experiment, it is statistically significant but the result is obviously not significant in the English-language meaning of the word.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#44

Post by bigfudge »

pfdietz wrote: post #38

There's a reason particle physicists want to see a five sigma effect before they believe a detection is real.

Start funding psychology etc like we fund physics and the problems would be much smaller. Most studies are underpowered. It’s not because scientists are lazy, it’s because they’re busy.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#45

Post by nestorD »

The paper "Frequentist and Bayesian inference: A conceptual primer" has some great things to say about the p-value and why a bayesian approach might solve some of the conceptual problems : https://www.researchgate.net/publication/326112369_Frequenti...

(Premise) If Tracy is an American then it is very unlikely that she is a US congresswoman; (Premise) Tracy is a US congresswoman; (Conclusion) It is very likely that Tracy is not an American

(Premise) If the H0 is true, then is it very unlikely that I will observe result X (Premise) I observe result X (Conclusion) Therefore, it is very likely that the H0 is not true

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#46

Post by omarhaneef »

kerkeslager wrote: post #16

If you're looking for a replacement you don't understand the problem. The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis . Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up he…

This is actually a much better explanation of the problem than the main article.

The main article kept saying that p values were not meant to be definitive, but without explaining what was wrong with them. At least, not very clearly.

This comment is a much clearer explanation -- imho -- as to what goes wrong in the industry. I also think it is right to focus the attention away from a particular statistical measure.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#47

Post by mkl »

bigfudge wrote: post #41
buboard wrote: post #29

Earlier quoted context omitted.

There are very few principles in science. Statistics tests seem to be something where scientists unanimouslu agree. Another is models, and applying occam's razor to choose the one with less parameters. Perhaps one could also justify choosing a hypothesis on the basis that it is simpler.

They certainly don’t agree. There has been 30+ years of objection to p < .05. The fact that many scientists are statistically illiterate doesn’t constitute consensus.

They pretty much agree that statistics is the tool to use, but disagree on the (important) details.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#49

Post by [deleted] »

bigfudge wrote: post #41
buboard wrote: post #29

Earlier quoted context omitted.

There are very few principles in science. Statistics tests seem to be something where scientists unanimouslu agree. Another is models, and applying occam's razor to choose the one with less parameters. Perhaps one could also justify choosing a hypothesis on the basis that it is simpler.

They certainly don’t agree. There has been 30+ years of objection to p < .05. The fact that many scientists are statistically illiterate doesn’t constitute consensus.

[deleted]

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#50

Post by inetknght »

blululu wrote: post #42

IMO the expression ‘statistical significance’ is a big part of the problem. Popular reporting of research translates a tiny but discriminable effect size into a SIGNIFICANT effect. Changing the nomenclature to ‘statistically discriminable’ would go a long way to improving popular understanding.

> Changing the nomenclature to ‘statistically discriminable’ would go a long way to improving popular understanding.

I think you overestimate the intelligence of people prone to misunderstanding. I don't think they're likely to understand big words like "statistically" or "discriminable" when they already don't understand big words like "statistical" or "significance".

Post reply on HN