51–60 of 142 posts
Earlier quoted context omitted.
It's different in that the cut-offs are communicated with careful wording, rather than a figure.
scientific papers communicate with numbers, errors, p-values next to numebrs, name of the statistical test used, and figures are usually annotated to indicate the level of significance. That's pretty careful.
The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.
Scientists have a duty to communicate what is true and what is not though. Passing that judgement duty to the general public is irresponsible and unwise (it's easy to claim anything with flawed / misleading statistics). Ideally someone would have come up with a framework that is better than statistic tests to justify levels of truth, but so far we dont have one
I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage. Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Whe…
I wrote an explanation for clinicians a while back that I people have had good luck with: http://madhadron.com/posts/2016-01-25-p_values_for_clinician...
> Likewise, we're supposed to assume that there is something magical about our prior assumptions?
No, we construct trials that a reasonable practitioner thinks satisfies the assumptions of the analysis used. And sometimes we learn that we had overlooked something and all those trials we did before we accounted for it were flawed.
If you're looking for a replacement you don't understand the problem. The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis . Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up he…
medicine is so ancient and people havent realised it yet. I cant wait for it to change
The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.
It's worth to point that "3% probability we're just seeing a pattern by accident" is only right when you understand it as "in the world here our hypothesis is wrong the same experiment would give such pattern in 3% cases", not as "given such result probability that we are wrong is 3%".
It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.
IMO the expression ‘statistical significance’ is a big part of the problem. Popular reporting of research translates a tiny but discriminable effect size into a SIGNIFICANT effect. Changing the nomenclature to ‘statistically discriminable’ would go a long way to improving popular understanding.
> Changing the nomenclature to ‘statistically discriminable’ would go a long way to improving popular understanding. I think you overestimate the intelligence of people prone to misunderstanding. I don't think they're likely to understand big words like "statistically" or "discriminable" when they already don't understand big words like "statistical" or "significance".
They do understand what "significant" means, they use the world every day. The problem is that it means something different than the intuitive every day meaning when used in context of p-values. Using a word like "discriminable" might help clear things up since it's a word that doesn't have have so much meaning packed into it already.
It's like when a mathematician says that something is "almost always" true, they mean something very different than when a non-mathematician says something is almost always true.
There's no way to guard against all false positives. And while changing the P value cutoff would reduce them, it would also increase false negatives. The answer doesn't lie in hard-line stances for or against P values or with an alternative that will have its own set of problems. It lies with greater education of those who run experiments & those who consumer the literature about proper interpretation and other metho…
Earlier quoted context omitted.
Cutoffs aren't the problem. They are inevitable; the logic is virtually identical to the question "Why can you drink at 21 and vote at 18?" Of course the Responsibility Fairy doesn't visit you that night and make you suddenly able to handle it when you weren't the day before, it's just that at scale, you don't have much choice but to operate that way because everything else is just too expensive. The problem with p-v…
> "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false? The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.
"Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used."
That's begging the question. The entire topic of conversation is whether or not the standards of justification are adequate.
I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage. Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Whe…
I'm interested in your perspective. What would be a better replacement for p?
I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change.
But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out.
EDIT:
To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use.
I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument.
But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question?
I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.