Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

fivethirtyeight.com

Post reply on HN
Print view View original

61–70 of 130 posts

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#61

Post by nonbel »

eanzenberg wrote: post #35
amluto wrote: post #21

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…

This is extremely important and I'm not surprised the media is either ignorant, doesn't understand, or 'stretches the truth' to adhere to their worldview. Statistical significance has NOTHING to do with magnitude of difference!

>"Statistical significance has NOTHING to do with magnitude of difference!"

Here is the equation for a t-statistic (used for the common t-test):

d=mean(a)-mean(b)

s=sqrt(var(a) + var(b))/sqrt(n)

t=d/s

I see the magnitude of difference (d) right in the numerator. The t-value is then compared to the t-distribution to get the tail probability.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#62

Post by thinkmoore »

btilly wrote: post #51
thinkmoore wrote: post #42

Earlier quoted context omitted.

Is this actually true for the Bayesian model? Throw in a parameter for whether or not Bill and Lorena were trying to have children until they had a boy and a girl. Now your answer will depend quite heavily on your prior!

This is entirely true. The posterior odds are computed ENTIRELY from the odds of the observed events under the prior beliefs. There is NO WAY in which might-have-beens and didn't-happens enter in. Therefore the posterior probabilities cannot depend on the knowledge of what they would have done if something different had happened. Of course frequentist statistics are heavily affected by what would have happened if som…

No, even as a Bayesian I could use either of two models to understand my data:

In world one, I use bayesian update on the model P(# boys | Bias) = Mutltinomial(n = 7, p = Bias)

In world two, I use bayesian update on the model P(# children | Bias) = Geometric(p = Bias)

Might-have-beens and didn't happens do play in, in my choice of model. I should choose the one that I believe, and if I'm not certain, I should use an even more complicated model that incorporates my beliefs about what models might be appropriate.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#63

Post by FrankyHollywood »

jules wrote: post #18
darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

That's not what a p-value is. A p-value is the probability of getting by random chance a result at least as extreme as the measurement. This is not the same as the probability that the effect you measured is due to chance. The latter isn't even well defined without additional assumptions.

Could you please explain what you mean, what is the difference you talk about?

Roll a dice 100 times, and on average 5 rolls have the pattern I want.

v.s. I have found pattern and there is a 5% change it was due to random.

Is that not the same?

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#64

Post by chc »

amluto wrote: post #21

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…

That's the second definition in Webster's 1913 edition. The first is:

> Fitted or designed to signify or make known something; having a meaning; standing as a sign or token; expressive or suggestive; as, a significant word or sound; a significant look.

It seems to me that this is the sense in which statisticians talk about significance. It means that the results actually signify something rather than just being meaningless noise.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#65

Post by statsaresimple »

chimeracoder wrote: post #2

For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article). Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether. Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas m…

I can't understand what is the problem with p-values. You should never talk about the results, the essential point is that the method provides valid conclusions 95% and invalid conclusions 5% of the time. If your conclusion is wrong (you are in the 5% part) that is not a paradox. Also you should not try to prove things since your conclusion can be wrong, you should be glad to have a procedure that gives you useful information but is not infallible. By being humble you solve the problem.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#66

Post by statsaresimple »

chimeracoder wrote: post #2

For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article). Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether. Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas m…

I can't understand what is the problem with p-values. You should never talk about the results, the essential point is that the method provides valid conclusions 95% and invalid conclusions 5% of the time. If your conclusion is wrong (you are in the 5% part) that is not a paradox. Also you should not try to prove things since your conclusion can be wrong, you should be glad to have a procedure that gives you useful information but is not infallible. By being humble you solve the problem.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#67

Post by FrankyHollywood »

carbocation wrote: post #10
eanzenberg wrote: post #5

5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.

> 5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so. On the other hand, let's sa…

Exactly right! There is a lot of misuse.

A friend of mine is researching cancer threatment at a commercial lab. They make a lot of long shots. Lot's of PHD students are hired, and they let them work on 1 of those longshots.

Costs them milions of euro's, but potentially gives them bilions.

These studies survive on statistics. It is even tweaked by having cross validations where they leave out the worst bins, 'because they are extremes'.

Always be very suspisious about statistical proofs, especially from companies with great commercial interests.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#68

Post by xtacy »

nonbel wrote: post #61
eanzenberg wrote: post #35

Earlier quoted context omitted.

This is extremely important and I'm not surprised the media is either ignorant, doesn't understand, or 'stretches the truth' to adhere to their worldview. Statistical significance has NOTHING to do with magnitude of difference!

>"Statistical significance has NOTHING to do with magnitude of difference!" Here is the equation for a t-statistic (used for the common t-test): d=mean(a)-mean(b) s=sqrt(var(a) + var(b))/sqrt(n) t=d/s I see the magnitude of difference (d) right in the numerator. The t-value is then compared to the t-distribution to get the tail probability.

You are also normalising the difference by the variance, so the t-statistic has no units.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#69

Post by wdewind »

skybrian wrote: post #59
wdewind wrote: post #49

Earlier quoted context omitted.

Right, but I guess what I'm getting at is frequently we see people doing even worse: making policy decisions based on data that doesn't even hit a minimum threshold for acceptability. I totally agree that people shouldn't make policy decisions solely because the data supports it, but frequently you see people make decisions based on data indicating something without actually being statistically significant. That feel…

One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all. So any minimum standard would need to take into account. Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the…

> One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all.

I mostly agree with this, but you never have that in a difficult decision.

> Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards?

Right! My entire point is that maybe data driven decision making isn't as useful as we think because it mostly doesn't hit scientific levels of accuracy. Maybe there are other, more intuitive systems, that make as much if not more sense, in absence of good data. How would we even know either way? We can't!

Regardless I'd love for the discussion to be "this is the best we can do" rather than "yes we did A/B testing so we know it's true!!!"

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#70

Post by shimn »

wdewind wrote: post #49
carbocation wrote: post #38

Earlier quoted context omitted.

More likely they mean the opposite: that a statistically significant P value (by whatever threshold you decide to use) should not be used, by itself, to drive policy decisions. Internally, the effect size still matters. Externally, there are numerous other factors that should drive decisionmaking.

Right, but I guess what I'm getting at is frequently we see people doing even worse: making policy decisions based on data that doesn't even hit a minimum threshold for acceptability. I totally agree that people shouldn't make policy decisions solely because the data supports it, but frequently you see people make decisions based on data indicating something without actually being statistically significant. That feel…

> should not be based only on whether a p-value passes a specific threshold.

It seems as though you'd missed the "only" in the clause. Further, they say nothing w.r.t. "does not pass" (aka: not hitting minimum threshold). Both your complaint and their point are orthogonal.

Post reply on HN