Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians want to abandon science’s standard measure of ‘significance’

sciencenews.org

Post reply on HN
Print view View original

101–110 of 142 posts

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#101

Post by DoofusOfDeath »

onlyrealcuzzo wrote: post #55
krcz wrote: post #28

Earlier quoted context omitted.

It's worth to point that "3% probability we're just seeing a pattern by accident" is only right when you understand it as "in the world here our hypothesis is wrong the same experiment would give such pattern in 3% cases", not as "given such result probability that we are wrong is 3%".

It seems like an education fail to me. Most people don't know the very basics of stats and we live in a world that's highly probabilistic. It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.

> It seems like an education fail to me.

I know a lot of people have trouble thinking clearly and correctly about probability and statistical inference.

But do we know if, practically speaking, that can be addressed by a modified educational curriculum?

Or are these concepts that would take an extraordinary amount of effort for many persons to understand well?

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#102

Post by DoofusOfDeath »

Can someone help me understand the concern about p-value hacking?

One of the comments below references this XKCD comic [0], which IIUC is an example of p-hacking.

But in that comic, the only difference I notice between the original hypothesis (jelly beans cause acne) and the p-hacked hypothesis (green jelly beans cause acne) is whether or not the hypothesis occurred to the researcher at the beginning of the study. And I don't understand why that would bear on the importance of each hypothesis.

[0] https://xkcd.com/882/

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#103

Post by btilly »

ghkbrew wrote: post #85
btilly wrote: post #77

Earlier quoted context omitted.

I disbelieve. The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs. Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly…

> But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences That's not how statistics works. Frequentist p-values and Bayesian inferences are both entirely dependent on the mathematical model in question. "Their plans if something else had happened" are a key part of the model. What you're describing at 2 entirely different experiments: 1) A couple has children…

No, that is EXACTLY how probability theory works.

The probability of event A given that we observed B is the probability of A and B happening divided by the probability that B happened. Mighta, coulda, shoulda but didn't doesn't enter into it and can't affect the result. The fact that frequentist statistics does care and shouldn't is one of the major criticisms that Bayesians offer.

If you think you understand statistics and don't understand this fact, then you do not understand statistics as well as you think you do. But you can be pardoned. Most statistics classes are too busy cramming statistical tests into student's heads to bother them with bothersome facts about where the cracks in the foundations are.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#104

Post by SquishyPanda23 »

CrazyStat wrote: post #76
SquishyPanda23 wrote: post #73

Earlier quoted context omitted.

The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute. Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way. You can actually compute p(H|x) if you use Bayesian stats.

With some caveats: the value you get for P(H|x) might be very sensitive to the priors you choose, and most people are not thoughtful enough about this.

Yes that is true, but there is a sense in which that just makes explicit a problem that already exists.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#105

Post by DataWorker »

just_steve_h wrote: post #40

This is a very old argument. I got my bachelor's degree in psychology at Harvard in 1993, and was told repeatedly that p-tests are abused, overused, and not terribly useful. To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p We were taught that "effect size" m…

Odd that in some ways psychology seems to be more informed, as a field, of these nuances and pitfalls and yet they have failed as a field to adequately police themselves. The problem isn’t on the statistics end of things, it’s the labor market and gatekeeping end of things. Another aspect of the inability to scale higher ed in the stupid way that many think is optimal.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#107

Post by wnoise »

Filligree wrote: post #97
patientplatypus wrote: post #34

I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage. Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Whe…

> Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Because that's how the math works. It isn't possible to compute a posterior without first having a prior, so you have to decide on a prior somehow . You can dress it up and try to hide it, but you can't avoid Bayes' theorem forever.

Well, you can report odds ratios (or their log).

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#108

Post by usgroup »

TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

I’m a Bayesian flavoured, and the “problem” IMO is that model driven statistics are really hard and beyond most scientists (people generally tbh).

If we required scientists to be good statisticians there’d be far fewer scientists.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#109

Post by kazinator »

TallGuyShort wrote: post #64
CreRecombinase wrote: post #27

Earlier quoted context omitted.

This is a a really common misconception about p values (that they can be interpreted as p(H0|x), or "probability of the null hypothesis given the data") when a p-value is in fact p(x|H0), or "probability of observing data at least this extreme given that the null hypothesis is true

Do you have a better of wording for inclusion in paper abstracts / article summaries than what I said? I'd love to hear one, but p(x|H0) is just as bad as p = 0.05.

"Our research creates a lot of confidence that the hypothesis is true. If our hypothesis happens to be false, then only 1 in 20 independent repetitions of our research (P = 0.05) will obtain the good results that we present here."

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#110

Post by vharuck »

roenxi wrote: post #43
TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

We use hard cutoffs for a bunch of things, they aren't perfect but they are fine. The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English. It is not only possible, but likely that people will be producing results that are insignificant but st…

I've been trying to not use the words "statistical significance" in my reports. It's hard to find alternatives, but "confident" usually fits the bill. Sadly, the reports need reviewed by other statisticians, and they much prefer I use the normal jargon because it's clearer to them.

But these concepts aren't clear to the people using the data. I've seen people use statistical tests done in batch without correcting for multiple testing (one of my regrettable tasks), and flat out ignore any differences that weren't statistically significant. They thought it meant "no difference."

With population-level data (e.g., Census counts or hospital records), I sometimes wonder if statistical significance has any value. It's a tool to make sure we don't accidentally claim a difference where there isn't one. But I can guarantee you that nothing worth measuring is exactly the same for men, women, whites, blacks, young, old, whatever. It'd be amazing if the difference was 0. So statistical significance ends up being an obfuscated way to state the sample size.

Post reply on HN