Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving to a World Beyond "p < 0.05" (2019)

tandfonline.com

Post reply on HN
Print view View original

11–20 of 111 posts

Re: Moving to a World Beyond "p < 0.05" (2019)

#11

Post by FollowingTheDao »

uniqueuid wrote: post #9
FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

The real underlying problem is that in your case, genetic variants are not accounted for. As soon as you include these crucial moderating covariates, it‘s absolutely possible to find true effects even for (rather) small samples (one out of a hundred is really to few for any reasonable design unless it‘s longitudinal)

> As soon as you include these crucial moderating covariates

Yes, but this is not usually done.

Re: Moving to a World Beyond "p < 0.05" (2019)

#12

Post by FollowingTheDao »

uniqueuid wrote: post #7
tomrod wrote: post #5

Earlier quoted context omitted.

From a methods perspective, wouldn't this be more of a statistical power issue (too small of sample size) than a random effect issue? Granted, we do a terrible job discussing statistical power.

The best way to talk about this is IMO effect heterogeneity. Underlying that you have the causal DAG to consider, but that‘s (a) a lot of effort and (b) epistemologically difficult!

> but that‘s (a) a lot of effort and (b) epistemologically difficult!

I agree, but then all these cheaper, easier studies are useless.

Re: Moving to a World Beyond "p < 0.05" (2019)

#13

Post by Incipient »

FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

Isn't that just a bad study? You have confounding factors - such as ethnicity - that weren't controlled/considered/eliminated.

I do get what you're saying, if you miss something in the study that is important, but I don't see how this is a case to drop the value of statistical significance?

Re: Moving to a World Beyond "p < 0.05" (2019)

#14

Post by uniqueuid »

FollowingTheDao wrote: post #11
uniqueuid wrote: post #9

Earlier quoted context omitted.

The real underlying problem is that in your case, genetic variants are not accounted for. As soon as you include these crucial moderating covariates, it‘s absolutely possible to find true effects even for (rather) small samples (one out of a hundred is really to few for any reasonable design unless it‘s longitudinal)

> As soon as you include these crucial moderating covariates Yes, but this is not usually done.

It is also not easy if you have many potential covariates! Because statistically, you want a complete (explaining all effects) but parsimonious (using as few predictors as possible) model. Yet you by definition don‘t know the true underlying causal structure. So one needs to guess which covariates are useful. There are also no statistical tools that can, given your data, explain whether the model sufficiently explains the causal phenomenon, because statistics cannot tell you about potentially missing confounders.

A cool, interesting, horrible problem to have :)

Re: Moving to a World Beyond "p < 0.05" (2019)

#15

Post by marcosdumay »

uniqueuid wrote: post #9
FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

The real underlying problem is that in your case, genetic variants are not accounted for. As soon as you include these crucial moderating covariates, it‘s absolutely possible to find true effects even for (rather) small samples (one out of a hundred is really to few for any reasonable design unless it‘s longitudinal)

Anything in health sciences has millions of variants not accounted for, that also interact between themselves so you'd need to account for every combination of them.

Re: Moving to a World Beyond "p < 0.05" (2019)

#16

Post by FollowingTheDao »

tomrod wrote: post #5
FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

From a methods perspective, wouldn't this be more of a statistical power issue (too small of sample size) than a random effect issue? Granted, we do a terrible job discussing statistical power.

Even if you did a study with the whole planet there would be no statistical significance since the genetic variation in in FADS genes are still in the minority. (The majority of the world is warm and this is a cold weather/diet adaptation).

In most African populations this Polymorphism does not exist at all. And even in Europeans it is only about 12% of the population.

Re: Moving to a World Beyond "p < 0.05" (2019)

#17

Post by uniqueuid »

FollowingTheDao wrote: post #12
uniqueuid wrote: post #7

Earlier quoted context omitted.

The best way to talk about this is IMO effect heterogeneity. Underlying that you have the causal DAG to consider, but that‘s (a) a lot of effort and (b) epistemologically difficult!

> but that‘s (a) a lot of effort and (b) epistemologically difficult! I agree, but then all these cheaper, easier studies are useless.

Unfortunately it‘s true that most studies are useless or even harmful (limited to certain disciplines).

Re: Moving to a World Beyond "p < 0.05" (2019)

#18

Post by fermisea »

FollowingTheDao wrote: post #11
uniqueuid wrote: post #9

Earlier quoted context omitted.

The real underlying problem is that in your case, genetic variants are not accounted for. As soon as you include these crucial moderating covariates, it‘s absolutely possible to find true effects even for (rather) small samples (one out of a hundred is really to few for any reasonable design unless it‘s longitudinal)

> As soon as you include these crucial moderating covariates Yes, but this is not usually done.

And it's usually discouraged by regulators because it can lead to p-hacking. I.e., with a good enough choice of control I can get anything down to 5%

The fundamental problem is the lack of embrace of causal inference techniques - i.e., the choice of covariates/confounders is on itself a scientific problem that needs to be handled with love

Re: Moving to a World Beyond "p < 0.05" (2019)

#19

Post by tomrod »

FollowingTheDao wrote: post #12
uniqueuid wrote: post #7

Earlier quoted context omitted.

The best way to talk about this is IMO effect heterogeneity. Underlying that you have the causal DAG to consider, but that‘s (a) a lot of effort and (b) epistemologically difficult!

> but that‘s (a) a lot of effort and (b) epistemologically difficult! I agree, but then all these cheaper, easier studies are useless.

Yes. The value is in metastudies and data fusion across more studies.

Re: Moving to a World Beyond "p < 0.05" (2019)

#20

Post by dleeftink »

> "I've become increasingly convinced that the idea of averaging is one of the biggest obstacles to understanding things... it contains the insidious trap of feeling/sounding "rigorous" and "quantitative" while making huge assumptions that are extremely inappropriate for most real world situations."

Semi-related, I found UniverseHacker's take[0] on the myth of averaging apt in regards to leaning to heavily on sample means. Moving beyond p-values and inter/intra group averages, there's fortunately a world of JASP[1].

[0]: https://news.ycombinator.com/item?id=41631448

[1]: https://jasp-stats.org/2023/05/30/jasp-0-17-2-blog/

Post reply on HN