Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians want to abandon science’s standard measure of ‘significance’

sciencenews.org

Post reply on HN
Print view View original

91–100 of 142 posts

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#91

Post by pfortuny »

SquishyPanda23 wrote: post #73
TallGuyShort wrote: post #64

Earlier quoted context omitted.

Do you have a better of wording for inclusion in paper abstracts / article summaries than what I said? I'd love to hear one, but p(x|H0) is just as bad as p = 0.05.

The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute. Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way. You can actually compute p(H|x) if you use Bayesian stats.

Wait wait wait: you need the priors, also. Where do they come from?

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#92

Post by mswen »

WhompingWindows wrote: post #84
TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

P-values are much more problematic at the alpha=.05 threshold than the alpha=.005 or alpha=.01 thresholds. In my work as a data scientist, 99% of the time this is not even relevant. We have huge sample sizes and most of the p-values we see are The question then: does a mean of 20.02 being significantly different from a mean of 20.03 over 10 million patients, is that actually important to us?

Yes, thank you. Practical significance. You cannot get away with just running statistical test or deep learning models. You often need to ask whether a change in product, service or process to take advantage of some trend or difference between customers segment will actually be worth doing. That involves getting your hands dirty in the business, trying to identify all the costs and side-effects of a change, and then how much of a size effect you will actually capture once the changes are made. My views in more detail [1]

[1] http://computationalimagination.com/article_practical_signif...

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#93

Post by riotman »

"More than 800 statisticians and scientists are calling for an end to judging studies by statistical significance in a March 20 comment published in Nature."

While their sources are supporting this statement, I'm getting mixed signals.

https://www.nature.com/magazine-assets/d41586-019-00857-9/da...

This is their primary source, of the statisticians calling to retire statistical significance. However, their primary reasoning is because statistics is misused to make erroneous conclusions. It seems like there is a lack of understanding about the philosophy and mathematics behind statistics that's the problem by its practitioners, not statistics itself.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#94

Post by marknutter »

buboard wrote: post #82
merpnderp wrote: post #52

Earlier quoted context omitted.

Richard Feynman said anything you can't prove yourself must be taken on pure faith and most of the public must take everything on pure faith as somehow they've left school without the faintest idea how basic stats work.

Feynman was talking to scientists. I do think that science has to provide answers, not just data. The public interprets every datum in a myriad of ways, and does not have a mechanism to establish consensus, hence you would end up with constant crisis.

Who says we're not in a constant crisis?

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#95

Post by andrewla »

CreRecombinase wrote: post #27
TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

This is a a really common misconception about p values (that they can be interpreted as p(H0|x), or "probability of the null hypothesis given the data") when a p-value is in fact p(x|H0), or "probability of observing data at least this extreme given that the null hypothesis is true

This is a more mathematically precise restatement of kerkeslager's explanation below and the obligatory xkcd [1]. The problem boils down to the fact that null results are not usually published, and that intuitive skepticism is not quantifiable.

Essentially, we want to get p(H0|x), but we need Bayes Law to get this from p(x|H0). But we need some notion of what priors to use. This is of course impossible to actually get, but if we published null studies then it would allow us to estimate it with better confidence. Alternatively we can ballpark it by saying how unexpected the result is, and how long the effect being explained has been studied.

The "common-sense" version of this is intuitive, that "extraordinary claims require extraordinary evidence". The more unexpected a result is, the lower the p-value has to be to be convincing.

The epidemiological version of this shows up in things like the Bradford Hill criteria [2], which include significance of association but also attempt to bring in plausibility.

[1] https://xkcd.com/882/

[2] https://en.wikipedia.org/wiki/Bradford_Hill_criteria

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#96

Post by andrewla »

SquishyPanda23 wrote: post #73
TallGuyShort wrote: post #64

Earlier quoted context omitted.

Do you have a better of wording for inclusion in paper abstracts / article summaries than what I said? I'd love to hear one, but p(x|H0) is just as bad as p = 0.05.

The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute. Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way. You can actually compute p(H|x) if you use Bayesian stats.

> You can actually compute p(H|x) if you use Bayesian stats.

You can't "compute" it for any useful meaning of the word "compute". You can estimate it intuitively, or you can try to look at how many pre-registered unpublished studies or null results have been published. Otherwise there's no way to even consider getting a grasp of what P(H) would be.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#97

Post by Filligree »

patientplatypus wrote: post #34

I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage. Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Whe…

> Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why?

Because that's how the math works. It isn't possible to compute a posterior without first having a prior, so you have to decide on a prior somehow.

You can dress it up and try to hide it, but you can't avoid Bayes' theorem forever.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#98

Post by SquishyPanda23 »

pfortuny wrote: post #91
SquishyPanda23 wrote: post #73

Earlier quoted context omitted.

The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute. Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way. You can actually compute p(H|x) if you use Bayesian stats.

Wait wait wait: you need the priors, also. Where do they come from?

That's definitely the hard part of switching to Bayesian methods. But the problem doesn't go away if you use p values, it just gets ignored.

Typically people doing research have prior information about what they are researching. E.g. previous studies have found effect sizes to be in some interval.

You can fallback on an uninformative prior or other tricks if you really want to model the idea that you know literally nothing about what you're researching. But that should be very rare.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#99

Post by grawprog »

onlyrealcuzzo wrote: post #55
krcz wrote: post #28

Earlier quoted context omitted.

It's worth to point that "3% probability we're just seeing a pattern by accident" is only right when you understand it as "in the world here our hypothesis is wrong the same experiment would give such pattern in 3% cases", not as "given such result probability that we are wrong is 3%".

It seems like an education fail to me. Most people don't know the very basics of stats and we live in a world that's highly probabilistic. It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.

I never learned statistics until my second year in university, it was really mind opening, especially with showing me just how much poorly done statistics with tiny sample sizes or misrepresented data are quoted and used as facts.

I actually found statistics pretty fun and eventually we learned to use R for it. Though I ended up missing 1 lecture and it almost screwed me over for the rest of the class. I found a Kahn academy video though that covered it and was able to catch up. There really is a ton of information to learn. It would have been nice to have at least touched on it a little bit in highschool beyond learning about means and the tiniest bit about standard deviations.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#100

Post by NohatCoder »

pfortuny wrote: post #91
SquishyPanda23 wrote: post #73

Earlier quoted context omitted.

The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute. Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way. You can actually compute p(H|x) if you use Bayesian stats.

Wait wait wait: you need the priors, also. Where do they come from?

You just make priors up, just like the 0.05 limit is made up. Making up a prior is basically asking the question: How likely does it seem that X has Y effect? Without having any data specific to that question.

Our 0.05 p-value limit is effectively just answering that question with a fixed 5%, no matter how ridiculous the proposition is.

Post reply on HN