Live data from Hacker News

Moving to a World Beyond "p < 0.05" (2019)

tandfonline.com

41–50 of 111 posts

Re: Moving to a World Beyond "p < 0.05" (2019)

#41
post #31

The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.

> The 0.05 threshold is indeed arbitrary, but the scientific method is sound.

I guess it depends on what you're referring to as the "scientific method. As the article indicates, a whole lot of uses of p-values in the field - including in many scientific papers - actually invoke statistics in invalid or fallacious ways.

Re: Moving to a World Beyond "p < 0.05" (2019)

#42

Earlier quoted context omitted.

In medicine, it is essentially impossible to control for all possible factors. Case in point, ethnicity is not biologically realized either; it's a social tool we use to characterize broad swathes of phenotypic [and sociocultural] differences that are more likely (but not guaranteed) to occur in certain populations. But the example provided of indigenous arctic people is itself imprecise. You can't control for that,…

> ethnicity is not biologically realized either What does this mean? Is it contrary to what OP is saying above?

Not exactly. What I mean to say is this: We know there are certain phenotypes that predominantly appear in certain populations, in broad strokes. But while we do have lists of correlates, we don't have good definitions for what an "ethnicity" is biologically, and there is very good reason to believe no satisfactory definition exists.

To use OP's example, we know that the gene mentioned is frequently found in the Inuit population. But if an Inuk does not have that gene, it does not somehow make them less Inuit. We can't quantify percentage Inuitness, and doing so is logically unsound. This is because the term "Inuit" doesn't mean its biological correlates. It simply has biological correlates.

To use an example of a personal friend, slightly anonymized: My friend is an Ashkenazi Jew. There is absolutely no uncertainty about this; Jewishness is matrilineal, and their mother was an Ashkenazi Jew, and her mother before her, going back over eight documented generations of family history. But alas - their grandfather was infertile, a fact that was posthumously revealed. Their maternal grandmother had a sperm donor. The sperm donor was not an Ashkenazi Jew. Consequently, can said friend be said to be "only 75% Jewish," having missed the "necessary" genetic correlates? Of course not. By simple matrilineage they are fully an Ashkenazi Jew.

Why are these terms used in medicine, then? Because, put simply, it's the best we can do. Genetic profiling is a useful tool under some limited circumstances, and asking medical subjects their ethnicity is often useful in determining medical correlates. But there is nothing in the gene that says "I am Inuk, I am Ashkenazi," because these ideas are social first, not genetic first.

Re: Moving to a World Beyond "p < 0.05" (2019)

#43
post #31

The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.

> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. I guess it depends on what you're referring to as the "scientific method. As the article indicates, a whole lot of uses of p-values in the field - including in many scientific papers - actually invoke statistics in invalid or fallacious ways.

The scientific method is sound != every experiment that claims to use the scientific method is sound

Re: Moving to a World Beyond "p < 0.05" (2019)

#44

"It is difficult to get a man to understand something, when his salary depends on his not understanding it." I am sure there are plenty of people who misunderstand or misinterpret statistics. But in my experience these are mostly consumers. The people who produce "science" know damn well what they are doing. This is not a scientific problem. This is a people problem.

> I am sure there are plenty of people who misunderstand or misinterpret statistics. But in my experience these are mostly consumers. The people who produce "science" know damn well what they are doing.

As a statistician, I could not disagree more. I would venture to say that most uses of statistics by scientists that I see are fallacious in some way. It doesn't always invalidate the results, but that doesn't change the fact that it is built on a fallacy nonetheless.

In general, most scientists actually have an extremely poor grasp of statistics. Most fields require little more than a single introductory course to statistics with calculus (the same one required for pre-med students), and the rest they learn in an ad-hoc manner - often incorrectly.

Re: Moving to a World Beyond "p < 0.05" (2019)

#45

Earlier quoted context omitted.

> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. I guess it depends on what you're referring to as the "scientific method. As the article indicates, a whole lot of uses of p-values in the field - including in many scientific papers - actually invoke statistics in invalid or fallacious ways.

The scientific method is sound != every experiment that claims to use the scientific method is sound

> The scientific method is sound != every experiment that claims to use the scientific method is sound

Sure, which is why I asked OP to define what they meant by "scientific method". The statement doesn't mean a whole lot if we're defining "scientific method" in a way that excludes 99% of scientific work that's actually produced.

Re: Moving to a World Beyond "p < 0.05" (2019)

#46
post #40

I think the main issue with "p < 0.05" can be summed up as people use it to "prove" a phenomenon exists rather than a pragmatic cutoff to screen for interesting things to investigate further.

Yeah but without a hardline how would you decide what to publish?

Re: Moving to a World Beyond "p < 0.05" (2019)

#47
post #40

I think the main issue with "p < 0.05" can be summed up as people use it to "prove" a phenomenon exists rather than a pragmatic cutoff to screen for interesting things to investigate further.

Yeah but without a hardline how would you decide what to publish?

Research is an institution. Just qualify the uncertainty and describe your further work to investigate.

Re: Moving to a World Beyond "p < 0.05" (2019)

#48

"It is difficult to get a man to understand something, when his salary depends on his not understanding it." I am sure there are plenty of people who misunderstand or misinterpret statistics. But in my experience these are mostly consumers. The people who produce "science" know damn well what they are doing. This is not a scientific problem. This is a people problem.

I haven't found this to be true at all. In fact, I'd say the majority of studies I read - even from prestigious journals - is fraught with bad statistics. I have no idea how some of these studies were even allowed to be published. Some fields are worse than others, but it's still a huge problem pretty much across the board.

People conduct science, and a lot of those people don't understand statistics that well. This quote from nearly 100 years ago still rings true in my experience:

"To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of."

- Ronald Fisher (1938)

Re: Moving to a World Beyond "p < 0.05" (2019)

#49
post #31

The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.

> A good researcher describes their study, shows their data and lays their own conclusions.

Tangent: I think that this attitude of scientific study can be applied to journalism to create a mode of articles between "neutral" reports and editorials. In the in-between mode, journalists can and should present their evidence without sharing their own conclusions, and then they should present their first-order conclusions (e.g. what the author personally thinks that this data says about reality) in the same article even if their conclusions are opinionated, but should restrain from second-order opinions (e.g. about what the audience should feel or do).

Re: Moving to a World Beyond "p < 0.05" (2019)

#50
post #35
post #31

The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.

> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. Agreed. A single published paper is not science, a tree data structure of published papers that all build off of each other is science.

Right, the decisions made by future researchers about what to base their work on are the real evaluation, hence citation counts as a core metric. It's easy to claim positive results by making your null hypothesis dogshit (and choice of "p" is easily the least inspired way to sabotage a null hypothesis), but researchers learn this game early and tend to not gamble their time following papers where they suspect this is what's going on. The whole thing kinda works, in a world where the alternatives don't work at all.
Post reply on HN