In undergrad, I learned how to about p-values but never quite understood how they were actually useful. Now, as a bioinformatics graduate student, I've come to understand that my original instinct was right all along.
Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
81–90 of 130 posts
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#82For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article). Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether. Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas m…
Where is the "paradox"? I assume that you mean that by pure hypothesis testing both Alice and Bob reject the null hypothesis at the alpha=0.05 level, for example. The fact that the p-value is irrelevant is by design: you perform the test and either you reject the null hypothesis or you don't. But in fact most people won't do "pure" hypothesis testing and will conclude (somewhat incorrectly) that the evidence against the null hypothesis is indeed stronger in Bob's experiment.
Edit: and why would be the answer to the second question "no"? The hypothesis testing procedure doesn't provide any point estimate at all so the question doesn't really mean anything in that setting.
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#83Earlier quoted context omitted.
You are also normalising the difference by the variance, so the t-statistic has no units.
The claim the significance has nothing to do with the magnitude of a difference is just wrong though. It clearly does, this information is just merged with other information about the variance and sample size to get the p-value, which is compared to a threshold to get significance.
it's not "just wrong". On the contrary, it is asymptotically true. In a world of abundant data, that matters.
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#84If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…
Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are). Whoa. Are you making a statement of what you think it would take to convince…
I'm making an intermediate claim. I find that most studies that claim "A does not cause B" to be extremely unconvincing exactly because of the statistical methodologies used and the fact that they don't actually do any statistics that would imply that A does not cause B.
The upshot is that I (and I'm more scientifically minded than most) am less than fully trusting of the interpretations given to a lot of claims resulting from scientific trials. Furthermore, I think a large part of this issue originates in both awful language used in actual papers and in the fact that people aren't doing the types of analysis that they should be doing.
The fact that I so frequently see news exclaiming some clinical result and that the statistical analysis (even if done rigorously and completely honestly) does not imply what the news claims means that something is very wrong in the way that statistics is used.
At least in Physics (my field), people tend to be a bit more careful about it. Physicists may do somewhat silly p-value tests, but they mostly insist on p values very very close to zero, and papers tend to at least answer the right question (such-and-such effect is upper-bounded by some tiny number to five sigmas).
I will backtrack a bit, though. I dug up "Vaccines are not associated with autism: An evidence-based meta-analysis of case-control and cohort studies". They do establish bounds on the odds ratio. Good for them.
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#85Earlier quoted context omitted.
No, even as a Bayesian I could use either of two models to understand my data: In world one, I use bayesian update on the model P(# boys | Bias) = Mutltinomial(n = 7, p = Bias) In world two, I use bayesian update on the model P(# children | Bias) = Geometric(p = Bias) Might-have-beens and didn't happens do play in, in my choice of model. I should choose the one that I believe, and if I'm not certain, I should use an…
Not if you follow Bayes' theorem. If you start with a prior distribution of beliefs about the likelihood of various ratios of boy vs girl births, the posterior distribution only depends on the observed outcomes. And the posterior distribution is exactly given by Bayes' theorem. One possible source of confusion for you is that Bayesian ideas have been a source of inspiration for a lot of ad hoc techniques (eg naive Ba…
Our goal is to calculate P(pb > 0.5 | "Six girls and one boy"), where pb is the probability of having a boy. (Ignoring that we have already assumed p is fixed), by applying Bayes' Theorem, we have:
P(pb > 0.5 | "Six girls and one boy") = (P("Six girls and one boy" | pb > 0.5) * P(pb > 0.5)) / P("Six girls and one boy")
Applying Bayes theorem thus requires us to calculate P("Six girls and one boy" | pb > 0.5). How do you suggest that we do this? Why is your answer the unique correct solution?
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#86Earlier quoted context omitted.
There are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.
I think the word "naive" is problematic here. Have you seen instances where Bayesians choose a prior that isn't at least somewhat informed by exploratory analysis?
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#87If people would just learn to resample (cross validate, use subsampling or the bootstrap), we wouldn't be having this pointless discussion at all.
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#88If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…
>Hint: doing big studies just so you can fail to find an effect is nonsensical. Instead, do big studies so you can put a tight upper bound on the effect. Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are).
Checking the bounds of a 100(1-a)% confidence interval is exactly equivalent to checking for a p-value below a%.
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#89If people would just learn to resample (cross validate, use subsampling or the bootstrap), we wouldn't be having this pointless discussion at all.
In theory, you are correct, but some data sets are too small or make it too difficult to cross validate in a way, that is meaningful to the original problem
Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
#90Earlier quoted context omitted.
That's not what a p-value is. A p-value is the probability of getting by random chance a result at least as extreme as the measurement. This is not the same as the probability that the effect you measured is due to chance. The latter isn't even well defined without additional assumptions.
Could you please explain what you mean, what is the difference you talk about? Roll a dice 100 times, and on average 5 rolls have the pattern I want. v.s. I have found pattern and there is a 5% change it was due to random. Is that not the same?
The things you're asking to compare aren't particularly clear, but I'm fairly certain they're not the same. The English phrase "due to" describes a causal connection between events, and p-values don't.