Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians want to abandon science’s standard measure of ‘significance’

sciencenews.org

Post reply on HN
Print view View original

131–140 of 142 posts

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#131

Post by bjourne »

just_steve_h wrote: post #40

This is a very old argument. I got my bachelor's degree in psychology at Harvard in 1993, and was told repeatedly that p-tests are abused, overused, and not terribly useful. To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p We were taught that "effect size" m…

> To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with pI don't think that is right. Think about tossing unbiased coins. The probability that you will get an improbable (p < 5%) result is exactly the same no matter how many or few coins you toss.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#132

Post by mar77i »

mokus wrote: post #116
danepowell wrote: post #31

Most comments here point to cherry picking and "p hacking" as being the primary problems with p values. Certainly those are major issues, but I think they miss the real point of the article, which is that null hypothesis testing is fundamentally broken, or at the very least doesn't do what most people think. A simple example of this can be shown with the following pair of tests: Testing for a fair coin: - Null hypoth…

The problem in the second case is not null-hypothesis testing, it’s sample size.

What about the fact that the probability of a non-American getting elected to congress being rather low?

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#133

Post by bjourne »

kerkeslager wrote: post #16

If you're looking for a replacement you don't understand the problem. The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis . Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up he…

> A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125.

Sally must be working using the wrong null hypothesis. If she attempts to reproduce Robert's results her null hypothesis should be whatever is the one he formulated. Perhaps that the probability of tails is at most 15%. Her p-value would be 2.66% (prob of seeing <= 2 heads) and if the world was fair, AJCFS would publish her study. But the world isn't and if Robert is a renowned scientist, significant+ is enough for him but failed replications of his work requires significant+++.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#134

Post by raizinho »

roenxi wrote: post #43
TallGuyShort wrote: post #14

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

We use hard cutoffs for a bunch of things, they aren't perfect but they are fine. The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English. It is not only possible, but likely that people will be producing results that are insignificant but st…

That's why the notion of practical significance exists expressed most frequently as an effect size.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#135

Post by MaxBarraclough »

bonoboTP wrote: post #26
Hitton wrote: post #11

Major problem with "significance" is that in some areas of research (say psychology), it's possible to gather lots of data(let 1000 people fill complex questionnaire) and from that data to fish for theory that's significant (in your sample Republicans might have been dumber than Democrats). But given the size of the sample and number of theories you test, you are bound to find something significant even if that isn't…

Many people talk about preregistration, but I'm not sure it would result in the hoped benefits. What I predict would happen: - either lots of studies are allowed to preregister, most of which cannot reject the null hypothesis. You end up with a lot of "boring" null-result papers in those high-profile journals that nobody gets excited about and nobody gets promoted for and no media coverage happens, bad marketing for…

> or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done

I see two things going on here.

Having additional 'oversight' for the scientific validity of a planned experiment, is presumably a good thing.

I don't see that it would be necessary to set out to prevent researchers investigating their intuitions.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#136

Post by dagw »

inetknght wrote: post #62
dagw wrote: post #56

Earlier quoted context omitted.

when they already don't understand big words like "statistical" or "significance". They do understand what "significant" means, they use the world every day. The problem is that it means something different than the intuitive every day meaning when used in context of p-values. Using a word like "discriminable" might help clear things up since it's a word that doesn't have have so much meaning packed into it already.…

> The problem is that it means something different than the intuitive every day meaning Yes that's my point. Using words that are hard to understand or only understandable in the context of P-values won't solve the problem; the problem being that the general public won't understand the subtle differences in meaning

I'm not sure. I think much of the confusion comes from people instinctively applying their every day understanding of the word "significance" and falsely believing they understand what it means in the statistical context. Had it been called statistical confliburance or some other made up term then they wouldn't think they knew what it meant and might find out what it actually meant.

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#137

Post by kerkeslager »

kqr wrote: post #130
kerkeslager wrote: post #61

Earlier quoted context omitted.

It seems clear to me that the author just didn't understand the problem with P values. It's part of a larger problem of science journalism being done by journalists without scientific backgrounds. It's not their fault--even if you have natural ability and interest in both communication and discovery, it's hard to get an education in both. I have the opposite problem: my abilities lie more in the statistics/science th…

Let me check if I get it right: can this be said to be a case of incorrectly aggregating experiments? We're in a sense taking min(p) over all p-values, or any(significant) over all results, when we should use an aggregation method that takes into account the total number of studies aggregated?

Yes.

For example, if you dig, you'll find that a lot of the evidence for a correlation between telomere shortening (associated with aging) and processed meat consumption comes from a 2008 study in the American Journal of Clinical Nutrition[1]. They found a P=.006 correlation between processed meat consumption and telomere shortening. But when you look into the study further, they collected data on 47 different food groups.

Let's calculate how unlikely a P=.006 event is if you try 47 times to find it:

A. If you have a probability of something happening once P_1, then the probability of that happening x times in a row, P_x, is P_x = P_1^x. For example, the probability of flipping a coin once and getting heads is P_1=.5, so the probability of that happening twice in a row is P_2=.5^2=.25, which you can verify by enumerating the possibilities (1 in 4 of the following combinations is 2 heads in a row: HH, HT, TH, TT). The probability of flipping 3 heads in a row is P_3=.5^3=.125 (only 1 of these 8 possibilities is all heads: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT).[2]

B. The chances of a P=.006 event NOT happening P' are P'=1-P=.994 (the rule of 1).

C. Combining facts A and B, the chances of a P_1=.006 event NOT happening (P'_1) 47 times in a row P'_47' is P'_47=.994^47=.754 (this is rounded to the 3 significant digits).

D. Applying the rule of 1 again, the probability of finding a P=.006 result if you try 47 times is P'_47' = 1-P'_47 = .246.

So basically, the actual confidence value on that study is P=.246: there's about a 1 in 4 chance they would find a P=.006 result for one of the 47 food groups tested if the null hypothesis is true. The null hypothesis in this case being "diet doesn't affect telomere length".

The paper doesn't seem to list their hypothesis, but they say things like "all others were P>.05", which indicates that they were hypothesizing a Pguaranteed that they would find a result within their definition of significance. This is very bad science, and a study with this bad of a statistical design never should have been funded in the first place.

This study has not been replicated as far as I know, but is often cited in pop science[3][4][5].

[1] https://academic.oup.com/ajcn/article/88/5/1405/4649028

[2] Note that this rule only applies to independent probabilities; the result of coin flip does not affect the result of the next coin flip, and the result of telomere correlation to one food group does not affect the result of telomere correlation to another food group. This rule won't work when applied to dependent probabilities, where the earlier tries affects later tries. For example if you draw an ace from a deck of cards (P=4/52) and don't place the ace back in the deck, the probability of drawing another ace is lower (P=3/51).

[3] https://www.livestrong.com/article/506649-foods-that-boost-t...

[4] https://resources.teloyears.com/diet-and-telomeres/processed...

[5] https://siimland.com/how-to-increase-telomere-length/

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#139

Post by Filligree »

wnoise wrote: post #107
Filligree wrote: post #97

Earlier quoted context omitted.

> Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Because that's how the math works. It isn't possible to compute a posterior without first having a prior, so you have to decide on a prior somehow . You can dress it up and try to hide it, but you can't avoid Bayes' theorem forever.

Well, you can report odds ratios (or their log).

That's good advice in general, but doesn't really answer the question of "How likely is it?".

Re: Statisticians want to abandon science’s standard measure of ‘significance’

#140

Post by kerkeslager »

bjourne wrote: post #133
kerkeslager wrote: post #16

If you're looking for a replacement you don't understand the problem. The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis . Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up he…

> A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125. Sally must be working using the wrong null hypothesis. If she attempts to reproduce Robert's results her null hypothesis should be whatever is the one he formulated. Perhaps that the probability of ta…

I had to think about this for a while to come up with a reasonable response. What you're saying seems a bit problematic to me.

The most obvious criticism that stands out to me is that when a control group is possible, that's where your null hypothesis should come from. But in the contrived coin-flipping example, there isn't a natural control, and in practice, many times control groups aren't available. So I don't know if that criticism actually applies to this situation.

The next thought I have is that if your goal is to determine significance (which should be the goal), rather than to find a significant result (which should not be the goal), using the same alternative and null hypotheses makes the math for aggregating the results easier. The power of replication is that it's multiplicative: you can get an aggregate P by multiplying the P values of aggregate studies. This means a P=.2 study can become significant if it's replicated (P=.2.2=.04), while a P=.03125 study can become lose significance with just one failure to reproduce (aggregating Robert and Sally's results, P=.03125.3125^-1=.1). I guess swapping the null and alternative hypotheses is a viable strategy for bringing attention to failures to replicate and it's not mathematically invalid, but that seems like changing your hypothesis to fix a problem with the academic publisher, and it complicates aggregating the P-values of replication studies.

Finally, it feels weird to me to use Robert's hypothesis as your null hypothesis. When Robert performed his experiment, his hypothesis was a stab in the dark due to a lack of pre-existing data (since he ostensibly didn't know about the other coin flipping experiments). But when attempting to reproduce a result, you have data, so it makes more sense to base your null hypothesis off the results of the previous experiment, not off the hypothesis of the previous experiment. As such, since Robert had 5 heads, the hypothesis would be that the probability of getting tails on a single coin flip is P=0, which gives Sally a VERY significant result. The numbers are weird because the experiment is contrived, but I think the general concept holds.

Post reply on HN