Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving to a World Beyond "p < 0.05" (2019)

tandfonline.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Moving to a World Beyond "p < 0.05" (2019)

#71

Post by dragonwriter »

FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

> So the researcher tests if omega 3 effects cardiovascular outcome in these hundred people by adding a lot more fish oil to the diet of these 100 people. Since only one of them really needs it, the P value will be insignificant and everyone will say fish oil does nothing. Yet for that one person it was literally everything.

But... that's not a problem with the use of the p-value, because that's (quite probably) a correct conclusion about the target (unrestricted) population addressed by the study as a whole.

That's a problem with not publishing complete observations, or not reading beyond headline conclusions to come up with future research avenues. That effects which are not significant in a broad population may be significant in a narrow subset (and vice versa) are well-known truths (they are the opposites of the fallacies of division and composition, respectively.)

Re: Moving to a World Beyond "p < 0.05" (2019)

#73

Post by [deleted] »

pif wrote: post #31

The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.

[deleted]

Re: Moving to a World Beyond "p < 0.05" (2019)

#74

Post by pif »

chimeracoder wrote: post #41
pif wrote: post #31

The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.

> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. I guess it depends on what you're referring to as the "scientific method. As the article indicates, a whole lot of uses of p-values in the field - including in many scientific papers - actually invoke statistics in invalid or fallacious ways.

> I guess it depends on what you're referring to as the "scientific method

No quotes needed, scientific method is well defined: https://en.wikipedia.org/wiki/Scientific_method

Re: Moving to a World Beyond "p < 0.05" (2019)

#75

Post by _yb2s »

pif wrote: post #31

The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.

As a scientist, I don’t think there is any specific scientific method or protocol - other than something really general like “think of all the ways people have been deceived in the past and carefully avoid them.” Almost no modern research follows anything like the “scientific method” I was taught in public school.

The way I do research is roughly Bayesian- I try to see what the aggregate of published experiments, anecdotes, intuition, etc. suggests are likely explanations for a phenomenon. Then I try to identify what realistic experiment is likely to provide the most evidence distinguishing between the top possibilities. There are usually many theories or hypotheses in play, and none are ever formally confirmed or rejected- only seen as more or less likely in the light of new evidence.

Re: Moving to a World Beyond "p < 0.05" (2019)

#76

Post by szundi »

I am a founder of a company managing 500k endpoints of different kind of tracking hardwares. It is funny how a thing that less likely to happen than once a month or year, like a bogus sample of data or an event lost gains significance when you have a client of just 200 endpoints who actually is interested in the service.

They just find mistakes, people come that their report for this guy was missing this and that and the commissions are not precise etc. Or they just get a bogus alert sometimes.

We could just shrug, but this erodes the trust about the whole service and the system itself. So we worked a lot to never have these. We have like 25 people on customer service and I can tell you it could have been easily 50 people otherwise.

The interesting part is how we had to kind of break people's common sense about what is significant problem and what is not. The "this is almost never happens I can assure you" from new colleagues were a good source of laughter after some time.

Re: Moving to a World Beyond "p < 0.05" (2019)

#77

Post by esafak »

FollowingTheDao wrote: post #3

As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…

That gets into Simpson's paradox; subpopulations can react differently than the whole.

https://en.wikipedia.org/wiki/Simpson%27s_paradox

Re: Moving to a World Beyond "p < 0.05" (2019)

#78

Post by mistermann »

iterance wrote: post #42
mistermann wrote: post #30

Earlier quoted context omitted.

> ethnicity is not biologically realized either What does this mean? Is it contrary to what OP is saying above?

Not exactly. What I mean to say is this: We know there are certain phenotypes that predominantly appear in certain populations, in broad strokes. But while we do have lists of correlates, we don't have good definitions for what an "ethnicity" is biologically, and there is very good reason to believe no satisfactory definition exists. To use OP's example, we know that the gene mentioned is frequently found in the Inui…

I don't disagree with this, but this is very not consistent with "ethnicity is not biologically realized", which suffers from the same logical error but in the other direction.

I often wonder how many entrenched culture battles could be ~resolved (at least objectively) by fixing people's cognitive variable types.

Re: Moving to a World Beyond "p < 0.05" (2019)

#79

Post by minkzilla »

johnobrien1010 wrote: post #53
tossandthrow wrote: post #47

Earlier quoted context omitted.

Research is an institution. Just qualify the uncertainty and describe your further work to investigate.

In THEORY yes, but in practice, there are not a ton of journals I think that will actually publish well done research that does not come to some interesting conclusion and find some p<.05. So....

That is what this article is about, changing the expectations of researchers and journals to be willing to publish and read research without p values.
Post reply on HN