Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving to a World Beyond "p < 0.05" (2019)

tandfonline.com

Post reply on HN
Print view View original

21–30 of 111 posts

Re: Moving to a World Beyond "p < 0.05" (2019)

#21

Post by iterance »

Incipient wrote: post #13
FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

Isn't that just a bad study? You have confounding factors - such as ethnicity - that weren't controlled/considered/eliminated. I do get what you're saying, if you miss something in the study that is important, but I don't see how this is a case to drop the value of statistical significance?

In medicine, it is essentially impossible to control for all possible factors. Case in point, ethnicity is not biologically realized either; it's a social tool we use to characterize broad swathes of phenotypic [and sociocultural] differences that are more likely (but not guaranteed) to occur in certain populations. But the example provided of indigenous arctic people is itself imprecise. You can't control for that, not without genetic testing - and even then, that presupposes we've characterized the confounding factors genetically, and that the confounding factors are indeed genetic in orgin at all.

Put another way, the population is simply too variable to attempt to eliminate all confounding factors. We can, at best, eliminate some of the ones we know about, and acknowledge the ones we can't.

Re: Moving to a World Beyond "p < 0.05" (2019)

#22

Post by zukzuk »

tomrod wrote: post #5
FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

From a methods perspective, wouldn't this be more of a statistical power issue (too small of sample size) than a random effect issue? Granted, we do a terrible job discussing statistical power.

Watching from the sidelines, I’ve always wondered why everything in the life sciences seems to assume unimodal distributions (that is, typically a normal bell curve).

Multimodal distributions are everywhere, and we are losing key insights by ignoring this. A classic example is the difference in response between men and women to a novel pharmaceutical.

It’s certainly not the case that scientists are not aware of this fact, but there seems to be a strong bias to arrange studies to fit into normal distributions by, for example, being selective about the sample population (test only on men, to avoid complicating variables). That makes pragmatic sense, but I wonder if it perpetuates an implicit bias for ignoring complexity.

Re: Moving to a World Beyond "p < 0.05" (2019)

#23

Post by staunton »

sva_ wrote: post #8
l5870uoo9y wrote: post #4

"Don’t base your conclusions solely on whether an association or effect was found to be “statistically significant” (i.e., the p-value passed some arbitrary threshold such as p Don’t believe that an association or effect exists just because it was statistically significant. Don’t believe that an association or effect is absent just because it was not statistically significant. Don’t believe that your p-value gives th…

> Hopefully this can help address the replication crisis[0] in (social) science. I think it isn't just p-hacking. I've participated in a bunch of psychology studies (questionaires) for university and I've frequently had situations where my answer to some question didn't fit into the possible answer choices at all. So I'd sometimes just choose whatever seems the least wrong answer out of frustration. It often felt lik…

> the study author's own beliefs and biases strongly influence how studies are designed

While studies should try to be as "objective" as possible, it isn't clear how this can be avoided. How can the design of a study not depend on the author's beliefs? After all, the study is usually designed to test some hypothesis (that the author has based on their prior knowledge) or measure some effect (that the author thinks exists).

Re: Moving to a World Beyond "p < 0.05" (2019)

#24

Post by tgv »

FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

Last I heard, 5 sigma was the standard for genetic studies now. pBut even though I'm not happy with NHST (the testing paradigm you describe), in that paradigm it is a valid conclusion for the group the hypothesis was tested on. It has been known for a long, long time that you can't find small, individual effects when testing a group. You need to travel a much harder path for those.

Re: Moving to a World Beyond "p < 0.05" (2019)

#25

Post by trabant00 »

sva_ wrote: post #8
l5870uoo9y wrote: post #4

"Don’t base your conclusions solely on whether an association or effect was found to be “statistically significant” (i.e., the p-value passed some arbitrary threshold such as p Don’t believe that an association or effect exists just because it was statistically significant. Don’t believe that an association or effect is absent just because it was not statistically significant. Don’t believe that your p-value gives th…

> Hopefully this can help address the replication crisis[0] in (social) science. I think it isn't just p-hacking. I've participated in a bunch of psychology studies (questionaires) for university and I've frequently had situations where my answer to some question didn't fit into the possible answer choices at all. So I'd sometimes just choose whatever seems the least wrong answer out of frustration. It often felt lik…

Psychology is IMO in the state alchemy was before chemistry. And there's no guarantee it will evolve beyond that. Not unless we can fully simulate the mind.

Re: Moving to a World Beyond "p < 0.05" (2019)

#26

Post by SkyBelow »

staunton wrote: post #23
sva_ wrote: post #8

Earlier quoted context omitted.

> Hopefully this can help address the replication crisis[0] in (social) science. I think it isn't just p-hacking. I've participated in a bunch of psychology studies (questionaires) for university and I've frequently had situations where my answer to some question didn't fit into the possible answer choices at all. So I'd sometimes just choose whatever seems the least wrong answer out of frustration. It often felt lik…

> the study author's own beliefs and biases strongly influence how studies are designed While studies should try to be as "objective" as possible, it isn't clear how this can be avoided. How can the design of a study not depend on the author's beliefs? After all, the study is usually designed to test some hypothesis (that the author has based on their prior knowledge) or measure some effect (that the author thinks ex…

There is a difference between a belief and an idea. I might have an idea about what causes some bug in my code, but it isn't a belief. I'm not trying to defend it, but to research it. Though I have met people who do hold beliefs about why code is broken. They refuse to consider the larger body of evidence and will cherry pick what we know about an incident to back their own view conclusions.

Can we recognize the beliefs we have that bias our work and then take action to eliminate those biases? I think that is possible when we aren't studying humans, but beliefs we have about humans are on a much deeper level and psychology largely doesn't have the rigor to account for them.

Re: Moving to a World Beyond "p < 0.05" (2019)

#27

Post by trabant00 »

"It is difficult to get a man to understand something, when his salary depends on his not understanding it."

I am sure there are plenty of people who misunderstand or misinterpret statistics. But in my experience these are mostly consumers. The people who produce "science" know damn well what they are doing.

This is not a scientific problem. This is a people problem.

Re: Moving to a World Beyond "p < 0.05" (2019)

#28

Post by biophysboy »

zukzuk wrote: post #22
tomrod wrote: post #5

Earlier quoted context omitted.

From a methods perspective, wouldn't this be more of a statistical power issue (too small of sample size) than a random effect issue? Granted, we do a terrible job discussing statistical power.

Watching from the sidelines, I’ve always wondered why everything in the life sciences seems to assume unimodal distributions (that is, typically a normal bell curve). Multimodal distributions are everywhere, and we are losing key insights by ignoring this. A classic example is the difference in response between men and women to a novel pharmaceutical. It’s certainly not the case that scientists are not aware of this…

It’s because statistical tests are based on the distribution of the statistic, not the data itself. If the central limit holds, this distribution will be a bell curve as you say

Re: Moving to a World Beyond "p < 0.05" (2019)

#29

Post by jerf »

FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

I was trawling studies for some issues of my own and sort of independently discovered this many years ago. It's very easy for an intervention to be life saving for 5%, pretty good for 10%, neutral for %84, and to have some horrible effect for %1, and that tends to average out to some combination of "not much effect", "not statistically significant", and depending on that 1% possible "dangerous to everyone". (Although with the way studies are run, there's a certain baseline of "it's super dangerous" you should expect because studies tend to run on the assumption that everything bad that happened during them was the study's fault, even though that's obvious not true. With small sample sizes this can not be effectively "controlled away".) We need some measure that can capture this outcome and not just neuter it away, because I also found there were multiple interventions that would have this pattern out outcome. Yet they would all be individually averaged away and the "official science consensus" was basically "yup, none of these treatments 'work'", resulting in what could be a quite effective treatment plan for some percentage of the population being essentially defeated in detail [1].

What do you mean? They all "work". None of them work for everyone, but that doesn't mean they don't work at all. As the case I was looking at revolved around nutritional deficiencies (brought on by celiac in my case) and their effects on the heart, it is also the case that the downside of the 4 separate interventions if it was wrong was basically nil, as were the costs. What about trying a simple nutritional supplement before we slam someone on beta blockers or some other heavy-duty pharmaceutical? I'm not against the latter on principle or anything, but if there's something simpler that has effectively no downsides (or very, very well-known ones in the cases of things like vitamin K or iron), let's try those first.

I think we've lost a great deal more to this weakness in the "official" scientific study methodology than anyone realizes. On the one hand, p-hacking allows us to "see" things where they don't exist and on the other this massive, massive overuse of "averaging" allows us to blur away real, useful effects if they are only massively helpful for some people but not everybody.

[1]: https://en.wikipedia.org/wiki/Defeat_in_detail

Re: Moving to a World Beyond "p < 0.05" (2019)

#30

Post by mistermann »

iterance wrote: post #21
Incipient wrote: post #13

Earlier quoted context omitted.

Isn't that just a bad study? You have confounding factors - such as ethnicity - that weren't controlled/considered/eliminated. I do get what you're saying, if you miss something in the study that is important, but I don't see how this is a case to drop the value of statistical significance?

In medicine, it is essentially impossible to control for all possible factors. Case in point, ethnicity is not biologically realized either; it's a social tool we use to characterize broad swathes of phenotypic [and sociocultural] differences that are more likely (but not guaranteed) to occur in certain populations. But the example provided of indigenous arctic people is itself imprecise. You can't control for that,…

> ethnicity is not biologically realized either

What does this mean? Is it contrary to what OP is saying above?

Post reply on HN