Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to Abolish "Statistical Significance"?

conversableeconomist.blogspot.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Time to Abolish "Statistical Significance"?

#31

Post by pathseeker »

reubens wrote: post #29
yosefzeev wrote: post #2

I suspect if you take away tenure being based upon publication, you will find that many statistical measures become more honest. You can abolish statistical significance, but it won't stop the abuse of knowledge, which is the real problem here anyway.

Could you expand on what you mean by the 'abuse of knowledge'? I agree that the focus on this metric negatively influences research outcomes, which extends to university structuring, but I'd like to hear your thoughts on how this extends to abuse of knowledge in general.

> Could you expand on what you mean by the 'abuse of knowledge'?

A nice way of saying "lying". Learning how to game the statistics (e.g. publish the 20th experiment that showed significance but fail to mention the other 19).

Re: Time to Abolish "Statistical Significance"?

#32

Post by mehrdadn »

reubens wrote: post #29
yosefzeev wrote: post #2

I suspect if you take away tenure being based upon publication, you will find that many statistical measures become more honest. You can abolish statistical significance, but it won't stop the abuse of knowledge, which is the real problem here anyway.

Could you expand on what you mean by the 'abuse of knowledge'? I agree that the focus on this metric negatively influences research outcomes, which extends to university structuring, but I'd like to hear your thoughts on how this extends to abuse of knowledge in general.

I think by knowledge they meant information asymmetry.

Re: Time to Abolish "Statistical Significance"?

#33

Post by nestorD »

One very important thing to know about the 0.05 threshold, and which I did not found in this thread, is that the ideal p-value for a problem is a function of the number of samples (and the effect size but it as a lesser impact).

0.05 is way to stringent if you have 10 samples and way too lenient if you have 1 million samples.

But, by force of convention, everyone is using 0.05 (a value suggested by Fischer when basically all datasets were small) independently of their sample size and in a world were we are sometimes reaching dataset size that would have been inconceivable when the threshold was suggested.

Here is a good article to start to think about how one would select a p-value for an experiment : https://journals.plos.org/plosone/article?id=10.1371/journal...

Re: Time to Abolish "Statistical Significance"?

#34

Post by ken666 »

In particle physics there is the concept of "look elsewhere" effect, precisely to take into account that if you look for a signal, for example of a particle of any mass in some range, there is the possibility that just by chance you find some statistical deviation at some mass value.

It is very different to confirm a prediction (i.e. to look for a particle with a precisely predicted mass), than to fish for some unexpected signal in your data.

In some cases Economics could do the same: Looking for an effect in any age range could be post-processed to take into account that you are looking into many age groups.

Re: Time to Abolish "Statistical Significance"?

#37

Post by nestorD »

ken666 wrote: post #34

In particle physics there is the concept of "look elsewhere" effect, precisely to take into account that if you look for a signal, for example of a particle of any mass in some range, there is the possibility that just by chance you find some statistical deviation at some mass value. It is very different to confirm a prediction (i.e. to look for a particle with a precisely predicted mass), than to fish for some unexp…

That's called the multiple comparaison problem in statistics and it is both well kown and compensated for in most studies (the hard part being not to have too mauch false negatives in an effort to keep the quantity of false positives constant) : https://en.wikipedia.org/wiki/Multiple_comparisons_problem

Re: Time to Abolish "Statistical Significance"?

#38

Post by no_identd »

Here's an alternative:

https://arxiv.org/abs/1904.06605

Victor Coscrato, Luís Gustavo Esteves, Rafael Izbicki, Rafael Bassi Stern — Interpretable hypothesis tests (2019)

Abstract:

Although hypothesis tests play a prominent role in Science, their interpretation can be challenging. Three issues are (i) the difficulty in making an assertive decision based on the output of an hypothesis test, (ii) the logical contradictions that occur in multiple hypothesis testing, and (iii) the possible lack of practical importance when rejecting a precise hypothesis. These issues can be addressed through the use of agnostic tests and pragmatic hypotheses.

Note that this enables acquiring one of the Holy Grail of Statistics, namely, controlling Type I & II errors simultaneously.

Re: Time to Abolish "Statistical Significance"?

#39

Post by cuchoi »

nestorD wrote: post #33

One very important thing to know about the 0.05 threshold, and which I did not found in this thread, is that the ideal p-value for a problem is a function of the number of samples (and the effect size but it as a lesser impact). 0.05 is way to stringent if you have 10 samples and way too lenient if you have 1 million samples. But, by force of convention, everyone is using 0.05 (a value suggested by Fischer when basic…

It is not a function of only the number of samples. It is also a function of how costly are false positives (type I error) and false negatives (type II error). That is (from my understanding) the paper's main point.

But then you have to be able to calculate how costly is a type I and a type II error! That's seems a relatively straightforward question for a business (for example in A/B testing), but how do you measure that cost in academia?

I think this would only introduce confusion and another variable for p-hacking.

Re: Time to Abolish "Statistical Significance"?

#40

Post by paulddraper »

kazinator wrote: post #11
paulddraper wrote: post #6

Earlier quoted context omitted.

It's as arbitrary as the age 18. Not a magic number, just a threshold with consensus.

If there is consensus, why are we seeing calls for abolishment from the scientific community?

Same with the age 18 :)

Historical consensus, then.

Post reply on HN