Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians want to abandon science’s standard measure of ‘significance’

sciencenews.org

Post reply on HN
Print view View original

31–40 of 142 posts

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#31

Post by danepowell »

Most comments here point to cherry picking and "p hacking" as being the primary problems with p values. Certainly those are major issues, but I think they miss the real point of the article, which is that null hypothesis testing is fundamentally broken, or at the very least doesn't do what most people think.

A simple example of this can be shown with the following pair of tests:

Testing for a fair coin:

  - Null hypothesis is you have a fair coin
  - You observe 100 heads in a row
  - Given a fair coin, it's extremely unlikely to observe 100 heads
  - Therefore it's not a fair coin
Okay, that makes sense, but this is logically the same as:

Testing whether a person "Bill" is an American:

  - Null hypothesis is Bill is an American
  - You observe *Bill is a US congressman*
  - Given Bill is an American, it's extremely unlikely to be a congressman
  - Therefore he's not an American
Obviously that's some broken logic, but it's a perfectly valid way to get p < .05

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#32

Post by bonoboTP »

mrkeen wrote: post #3

tl;dr: > Is there a better way to judge if a study is solid? > Unfortunately, there is no single alternative that everyone agrees would be better for all experiments.

How about understanding and valuing a publication as a letter to the research community about something the authors think is interesting and needs further consideration. The start of a scientific debate. A "hey what do you other research groups around the world think about this?" If nobody thinks anything and it doesn't gain traction or followup debate/research/consideration, then it failed its mission.

Not as "This research has now found X and is in a reputable journal, so this is the new state of science from now on".

The problem is, academics are pushed to writing more and more papers. There's no time to discuss back and forth too much about an already published paper, there are new papers to write! The goal of the project was to publish a paper, published=won, end of story, next project on.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#33

Post by adj83 »

This gets at the heart of fake science. Everyone today thinks surveys or studies are science.

The purpose of a paper is supposed to be to contribute something new to a field, not just perform a statistical study.

But you or I cant say that now as so many organisations use "studys" as the basis for their field while maintaining that they are scientific.

For me, the fact this is even being discussed really is good news.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#34

Post by patientplatypus »

I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage.

Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Where is it written that our null and alternative hypothesis should conform to what is true, when our best ability to predict the future (as people) is based on our past experience? The world doesn't always behave predictably, and yet that seems to be baked into the assumption of every regression test ever run. Completely bonkers if you ask me.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#35

Post by TrinaryWorksToo »

patientplatypus wrote: post #34

I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage. Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Whe…

I'm interested in your perspective. What would be a better replacement for p?

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#36

Post by turk73 »

kerkeslager wrote: post #16

If you're looking for a replacement you don't understand the problem. The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis . Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up he…

This is interesting because when I learned all this stuff the P value wasn't the only measure of study design quality. There was also statistical power which here would related to number of coin flips performed. You simply can't flip a coin 5 times and say you have statistically valid results you would be laughed out of the auditorium. Maybe it requires 1000 independent trials, or 10,000, but certainly not 5. The P value was just about confidence level of rejecting the null hypothesis, but much more has to go into study design in order to make it valid.

I have not done this sort of work since the early 2000s, I am relying on memory and a couple of quick web searches.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#37

Post by ineedasername »

There's no way to guard against all false positives. And while changing the P value cutoff would reduce them, it would also increase false negatives.

The answer doesn't lie in hard-line stances for or against P values or with an alternative that will have its own set of problems. It lies with greater education of those who run experiments & those who consumer the literature about proper interpretation and other methods of analysis that should be used along side it.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#39

Post by ludamad »

TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

"3% probability we're just seeing a pattern by accident" in fact still sounds a lot better than it is if there is a positive result bias, as you'll miss the "but 99 percent of scientists who attempted this did not see a pattern"

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#40

Post by just_steve_h »

This is a very old argument. I got my bachelor's degree in psychology at Harvard in 1993, and was told repeatedly that p-tests are abused, overused, and not terribly useful.

To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with pWe were taught that "effect size" measures were crucial to understanding and interpreting results. If you have p<.05 but a tiny effect size, you're likely not seeing a meaningful difference.

Post reply on HN