Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

fivethirtyeight.com

Post reply on HN
Print view View original

71–80 of 130 posts

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#71

Post by statsaresimple »

amluto wrote: post #21

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…

To solve the mystery of the significance of the p-value:

If mankind were to make one unique experiment a p-value would be a useful tool, but if many experiments are realized and people hide the results only showing those results that they want to show for self-promotion, then selecting a part of the real information (purposely or not) is a lie.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#72

Post by amluto »

chc wrote: post #64
amluto wrote: post #21

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…

That's the second definition in Webster's 1913 edition. The first is: > Fitted or designed to signify or make known something; having a meaning; standing as a sign or token; expressive or suggestive; as, a significant word or sound; a significant look. It seems to me that this is the sense in which statisticians talk about significance. It means that the results actually signify something rather than just being meani…

Interesting. That definition seems like a bit of a stretch in this context to me. Results of trials aren't "fitted or designed" to signify -- they are or are not significant by the p-value standard, and whether they are or are not is random (which is the whole point).

In any event, I suspect that, among most currently living English speakers, the second definition is what comes to mind.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#73

Post by snowwrestler »

wdewind wrote: post #49
carbocation wrote: post #38

Earlier quoted context omitted.

More likely they mean the opposite: that a statistically significant P value (by whatever threshold you decide to use) should not be used, by itself, to drive policy decisions. Internally, the effect size still matters. Externally, there are numerous other factors that should drive decisionmaking.

Right, but I guess what I'm getting at is frequently we see people doing even worse: making policy decisions based on data that doesn't even hit a minimum threshold for acceptability. I totally agree that people shouldn't make policy decisions solely because the data supports it, but frequently you see people make decisions based on data indicating something without actually being statistically significant. That feel…

> That feels like a bigger problem to me than people actually getting good data but then using it too confidently

P value alone does not tell you whether you have good data. It just tells you how well a particular model fits the data that you have. It won't (and can't) tell you if your data set is missing data that would alter the P value were it to be included.

P value alone is not enough to say "good" or "bad." That's why this statement matters.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#74

Post by achompas »

btilly wrote: post #52
yummyfajitas wrote: post #41

Earlier quoted context omitted.

Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different. I don't think this is a completely accurate portrayal of Bayesian stats. In Bayesian stats, there is no "real prior". Probability distributions are all subjective representations o…

There are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.

I think the word "naive" is problematic here. Have you seen instances where Bayesians choose a prior that isn't at least somewhat informed by exploratory analysis?

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#75

Post by cf »

amluto wrote: post #21

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…

This is a fantastic point. Sometimes when someone hands me a study I will ask what is the effect size. In some studies, even if there was a discernible effect, there is no hope for it to be anything but a small effect.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#76

Post by cschmidt »

achompas wrote: post #74
btilly wrote: post #52

Earlier quoted context omitted.

There are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.

I think the word "naive" is problematic here. Have you seen instances where Bayesians choose a prior that isn't at least somewhat informed by exploratory analysis?

A uniform prior would be one example. That's just saying "I've got no idea what the value is".

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#77

Post by btilly »

thinkmoore wrote: post #62
btilly wrote: post #51

Earlier quoted context omitted.

This is entirely true. The posterior odds are computed ENTIRELY from the odds of the observed events under the prior beliefs. There is NO WAY in which might-have-beens and didn't-happens enter in. Therefore the posterior probabilities cannot depend on the knowledge of what they would have done if something different had happened. Of course frequentist statistics are heavily affected by what would have happened if som…

No, even as a Bayesian I could use either of two models to understand my data: In world one, I use bayesian update on the model P(# boys | Bias) = Mutltinomial(n = 7, p = Bias) In world two, I use bayesian update on the model P(# children | Bias) = Geometric(p = Bias) Might-have-beens and didn't happens do play in, in my choice of model. I should choose the one that I believe, and if I'm not certain, I should use an…

Not if you follow Bayes' theorem. If you start with a prior distribution of beliefs about the likelihood of various ratios of boy vs girl births, the posterior distribution only depends on the observed outcomes. And the posterior distribution is exactly given by Bayes' theorem.

One possible source of confusion for you is that Bayesian ideas have been a source of inspiration for a lot of ad hoc techniques (eg naive Bayes) which do NOT follow Bayesian rules of inference. The reason is that exact inference in Bayes nets is NP-hard. So you're used to hearing "Bayesian" applied to things that have nothing to do with Bayes' formula.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#78

Post by nonbel »

xtacy wrote: post #68
nonbel wrote: post #61

Earlier quoted context omitted.

>"Statistical significance has NOTHING to do with magnitude of difference!" Here is the equation for a t-statistic (used for the common t-test): d=mean(a)-mean(b) s=sqrt(var(a) + var(b))/sqrt(n) t=d/s I see the magnitude of difference (d) right in the numerator. The t-value is then compared to the t-distribution to get the tail probability.

You are also normalising the difference by the variance, so the t-statistic has no units.

The claim the significance has nothing to do with the magnitude of a difference is just wrong though. It clearly does, this information is just merged with other information about the variance and sample size to get the p-value, which is compared to a threshold to get significance.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#79

Post by jkyle »

darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

I'm probably jumping into shark filled waters considering the point of the article is that defining p-values intelligibly is incredibly difficult even for experts, but here goes....

> Is the p-value really not the probability of your results being due to chance?

No, it is not. It is the probability that the statistical attributes of the data would be equal or more extreme than observed assuming the Null Hypothesis is true.

In your definition, there is no assumption of what model the data should conform to...so what does "by random" mean in that context? Also, by random doesn't mean 'not predictable' or useless. If I roll a 100 sided die an infinite number of times, I'd expect the number of observances of '5' to approach .01 of the total distribution. So, the probability of rolling a 5 by random chance is 1 in 100. However, I would not reject my Null Hypothesis since my model predicts exactly this random behavior.

Now, if I rolled a die 1000 times and rolled a 5 every time the mean of that distribution (5) would be very, very far from the expected mean of my model if the Null is assumed to be true. And I may be tempted (very) to reject the Null that I am rolling a 100 sided fair die.

I will now sit back and wait for my definition and analogy to be torn to shreds. :)

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#80

Post by tokenadult »

amluto wrote: post #21

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…

Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are).

Whoa. Are you making a statement of what you think it would take to convince a hard-line anti-vaccination activist (the most charitable interpretation I can think of for your statement), or are you making that statement on your own behalf? I am chagrined to see this statement in the top comment in this thread. Reading the fine article submitted for discussion is strongly encouraged here on Hacker News, not just reading the headline, and I think you may have missed the part of the article that says, "Indeed, many of the ASA committee’s members argue in their commentaries that the problem isn’t p-values, just the way they’re used — 'failing to adjust them for cherry picking, multiple testing, post-data subgroups and other biasing selection effects,' as Deborah Mayo, a philosopher of statistics at Virginia Tech, puts it. When p-values are treated as a way to sort results into bins labeled significant or not significant, the vast efforts to collect and analyze data are degraded into mere labels, said Kenneth Rothman, an epidemiologist at Boston University."

Discernibility of effects versus lack of discernibility of effects is not the main issue here. The main issue is what effects are even possible, plausible, or likely in the first place. One way in which the law of evidence as it has developed in legal procedure over centuries is stronger than the statistical processes used in scientific publications is that the law of evidence remembers that human beings can have biases. I am surprised to see a participant here on Hacker News holding to the strong prior that we must assume that vaccines have caused some cases of autism. Nope. There has never, ever, ever, been any evidence that suggests a causal mechanism such that vaccines could cause even one case of autism. Moreover, there is abundant evidence that the first persons who promoted such claims were trying to abuse the tort law system in England to shake down vaccine manufacturers for payments for imaginary harms.[1] The kind of thinking brought into the discussion here in the comment quoted above doesn't take into account the difference between "evidence-based medicine" and "science-based medicine,"[2] in which clinical trials and other statistical data are supplemented with information from basic science about what priors are plausible and which are plainly implausible. There is no effect discernible from vaccines in increasing rates of autism because there is, on the basis of multiple lines of evidence, no effect that is the tiniest bit plausible for such a claimed harm, and a very plausible line of evidence showing greed and desire for publicity on the part of the corrupt lawyers and doctors who introduced the claim.

[1] http://www.bmj.com/content/342/bmj.c5347

[2] https://www.sciencebasedmedicine.org/about-science-based-med...

Post reply on HN