Why is 0.05 so common to begin with. Its an arbitrary value ain't it?
It's as arbitrary as the age 18. Not a magic number, just a threshold with consensus.
If there is consensus, why are we seeing calls for abolishment from the scientific community?
Time to Abolish "Statistical Significance"?
11–20 of 41 posts
Re: Time to Abolish "Statistical Significance"?
#12Earlier quoted context omitted.
Yep. But if you're going to make binary publishing decisions then you have to draw a line somewhere.
However "1/20 chance that, if the hypothesis is false, our research confirms it anyway" is not a good choice of somewhere for any important research that can affect people's lives.
What would you suggest, and what contexts do you think .05 would be appropriate for, if any?
Re: Time to Abolish "Statistical Significance"?
#13That was a lot of words. There are two alternatives to the current methodology: * Remove significance requirement for publishing * Adopt another statistical measure like Bayesian stats
Maybe it would be a good idea to have an exception where an academic paper with a p-value that's right on the borderline can get published if he or she can find someone at another institution to replicate their results. At the very least it would ensure that someone else is reproducing their methodology.
There's a potential for abuse (as always), but perhaps you could curb it by mandating that you can only use the same co-submitter once. The other (likely bigger) problem would be figuring out how to get funding for it.
Full disclosure, I've never worked in a academia before, so please take this all with a grain of salt.
Re: Time to Abolish "Statistical Significance"?
#14Earlier quoted context omitted.
Yep. But if you're going to make binary publishing decisions then you have to draw a line somewhere.
However "1/20 chance that, if the hypothesis is false, our research confirms it anyway" is not a good choice of somewhere for any important research that can affect people's lives.
This debate isn't that simple. The other side of this isn't that everything magically just gets more accurate. The cost will be withholding true discoveries from the medical community for longer.
Imagine you've done a study of something else and you accidentally discover a (properly FDR controlled) correlation between some drug and heart attacks, significant at 0.03 and with no big contrary prior.
Should you publish or wait 5 years for a follow-up study to complete?
What if three different groups stumble across this same finding? Should they all publish and maybe someone will realize 'shit this drug is killing people' or should they all wait to hit some other higher standard of significance?
The point is theres always a balance between specificity and sensitivity, a tradeoff in terms of costs.
I'd personally be happy with keeping 0.05 as the threshold for 'probably something here'. The real issue are about publication bias, incentives, naive interpretation of published work 'heres one study in Nature so it must be true'. I don't see any other purely statistical change, Bayesian (essentially has all the same problems, aside from taking into account priors) or whatever, that will solve these in any way that won't come without unacceptable sensitivity cost.
Re: Time to Abolish "Statistical Significance"?
#15Who is Timothy Taylor? (The 'about me' on his Blogger page is blank, and Googling doesn't turn up results that are obviously about him.) This refrain is so well worn by now that it seems that one really ought to have something fundamentally new to say, or be so much of a heavy weight that one's own opinion might be enough significantly to swing the pendulum, before hoping to have another repetition of it make any significant difference. (On the other hand, I guess it's just a blog post, so I shouldn't spend too much energy ranting about how someone uses his own blog.)
Re: Time to Abolish "Statistical Significance"?
#16That was a lot of words. There are two alternatives to the current methodology: * Remove significance requirement for publishing * Adopt another statistical measure like Bayesian stats
How many hypothesis testing methods are there? So far I've seen only two.
Re: Time to Abolish "Statistical Significance"?
#17Earlier quoted context omitted.
It's as arbitrary as the age 18. Not a magic number, just a threshold with consensus.
If there is consensus, why are we seeing calls for abolishment from the scientific community?
I think you might be conflating two related-but-different arguments.
There is consensus that a p-value of 0.95 is reasonable. No one is arguing against that parameter in particular.
The argument is rather against the idea of statistical significance itself, because it's relatively easy to cheat if you are dishonest. Setting p to 0.01 won't change that.
Finally, and this is my personal opinion, there is "consensus" for abolishing statistical significance the same way there's "consensus" that Python is the best programming language. Scientists are more or less in agreement that it could be better, but "mildly displeased scientists call for unclear improvement" is not as good a headline.
Re: Time to Abolish "Statistical Significance"?
#18Why is 0.05 so common to begin with. Its an arbitrary value ain't it?
Yep. But if you're going to make binary publishing decisions then you have to draw a line somewhere.
There have been some recent proposals to lower it getting lots of traction. Sure if you want to decide whether to publish based on p-value alone you have to draw a line somewhere, but 0.05 isn't necessarily the right place. And you don't make publishing decisions based on p-values alone.
Re: Time to Abolish "Statistical Significance"?
#19Earlier quoted context omitted.
If there is consensus, why are we seeing calls for abolishment from the scientific community?
I think you might be conflating two related-but-different arguments. There is consensus that a p-value of 0.95 is reasonable. No one is arguing against that parameter in particular. The argument is rather against the idea of statistical significance itself, because it's relatively easy to cheat if you are dishonest. Setting p to 0.01 won't change that. Finally, and this is my personal opinion, there is "consensus" fo…
[deleted]
Re: Time to Abolish "Statistical Significance"?
#20Earlier quoted context omitted.
If there is consensus, why are we seeing calls for abolishment from the scientific community?
I think you might be conflating two related-but-different arguments. There is consensus that a p-value of 0.95 is reasonable. No one is arguing against that parameter in particular. The argument is rather against the idea of statistical significance itself, because it's relatively easy to cheat if you are dishonest. Setting p to 0.01 won't change that. Finally, and this is my personal opinion, there is "consensus" fo…
There is not consensus that the current threshold is reasonable. There are many big names arguing against that parameter in particular.
See e.g. Redefine statistical significance. The authors list is practically a who's who https://psyarxiv.com/mky9j
>One Sentence Summary: We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.