Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Psychology Journal Bans Significance Testing

sciencebasedmedicine.org

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Psychology Journal Bans Significance Testing

#41

Post by Al-Khwarizmi »

Al-Khwarizmi wrote: post #40
rcthompson wrote: post #15

I spent about a hour explaining p-values to a fellow graduate-level researcher a few weeks ago. I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it. In undergrad, hypothesis testing was more or less taught as "do this inscrutable calculation and if the result is 0.05 or less, you win". The point is, in my experience, a lot of people really don't get p-values, e…

I understand p-values, but I always have real problems understanding the thing of 95% confidence interval not meaning 95% probability of the true parameter being in the interval. I once grasped it, but then I forgot the reason. And now I look at this paragraph: "the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interv…

OK, I think I more or less got it and I can answer my own question... correct me if I'm wrong, but I think the interpretation "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?" is flawed because we are not talking about the same interval every time, right?

If I have a variable that is always positive, then I could have a weird procedure to generate confidence intervals that gives me the interval [-inf,0] 5% of the time and the interval [0,inf] 95% of the time.

This would meet the definition of confidence intervals perfectly, and yet when I get [-inf,0] the real probability of the parameter being in the interval is 0%, and when I get [0,inf], it's 100%.

I wonder how large this discrepancy may be in practice (as this is obviously a made-up extreme case).

Re: Psychology Journal Bans Significance Testing

#42

Post by algorias »

yummyfajitas wrote: post #39
algorias wrote: post #38

Earlier quoted context omitted.

> The fact is that to make a good decision, eventually you need to compute a single number. Your linked blog post states that if you make a good decision, then there is a process computing a single number which is equivalent to your process. This is not equivalent to what you claim. As a matter of fact, it's the same kind of confusion that exists around the p-value. It's not the case that a process explicitly computi…

Eventually you need to compute a number which is either above or below your go/no go threshold. That's the number I'm referring to. I don't claim you can't arrive at it by some perfect heuristic. I merely claim that you are better off being explicit about your assumptions and formalizing your reasoning. That just makes mistakes more obvious, makes your strong assumptions more clear, and makes it more likely that you…

I see where you're coming from, and I agree with you in large part, specially about making your assumptions explicit.

However, I think it's important to notice that an explicit formula for your thought processes can be difficult (computationally expensive) to find. Our brains have evolved to use heuristics and "gut feelings" to make decisions, and the approach you propose forces you to throw all that away and use the much slower general purpose processing part of your brain to emulate those processes. So there's a tradeoff there.

Re: Psychology Journal Bans Significance Testing

#43

Post by stdbrouw »

Al-Khwarizmi wrote: post #40
rcthompson wrote: post #15

I spent about a hour explaining p-values to a fellow graduate-level researcher a few weeks ago. I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it. In undergrad, hypothesis testing was more or less taught as "do this inscrutable calculation and if the result is 0.05 or less, you win". The point is, in my experience, a lot of people really don't get p-values, e…

I understand p-values, but I always have real problems understanding the thing of 95% confidence interval not meaning 95% probability of the true parameter being in the interval. I once grasped it, but then I forgot the reason. And now I look at this paragraph: "the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interv…

"The probability of this parameter being in the interval" is what's called a credible interval, and it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely.

In general, frequentists just don't like to talk about the properties of this particular sample, only about long-term frequencies – hence the name. Why? Because they object to the idea of probability as a degree of belief rather than as an objective measure, and given that attitude the statement that "there's a 95% probability the parameter is in this interval" doesn't make any sense: either it's in the interval or it isn't.

Re: Psychology Journal Bans Significance Testing

#44

Post by stdbrouw »

Banning p-values makes sense to me, as they force you to declare an effect as either significant or not significant, rather than looking at the preponderance of the evidence and building knowledge over multiple experiments. It also leads us to focus on statistical uncertainty at the expense of all of the other kinds of uncertainty researchers are faced with: do I have the theory to back this up, am I actually measuring what I am trying to measure, is this coherent with other findings in the field? I do think the editors might be right when they say banning p-values will make the quality of the research go up, not down.

But if you read the original editorial, up at http://www.tandfonline.com/doi/pdf/10.1080/01973533.2015.101..., you can see that they also reject confidence intervals and Bayesian reasoning with uniform priors (which is really the same thing) without providing any guidance at all on better procedures. I fear that will just lead readers to try and guess the reliability of the data themselves, or worse, interpret the sample statistics as numbers without any associated uncertainty.

So they're doing away with poor statistical procedures, but at what cost? It's like that old joke: we've found a 100% reliable cure for cancer – bombing the planet until everybody's dead.

Re: Psychology Journal Bans Significance Testing

#45

Post by stdbrouw »

CountBayesie wrote: post #13

>The type of analysis being banned is often called a frequentist analysis I find that there is a trend of associating "bad statistics" with "Frequentists Statistics" which isn't really fair. If you found a statistician trained only in Frequentist methods and asked their opinion on experiment design in psychological research they would likely be just as appalled as any Bayesian. I'm a big fan of Bayesian methods, but…

The problem is that p-values are begging to be misunderstood, and in fact you cannot use them as a decisionmaking procedure without "misinterpreting" them – after all, you're deciding whether to accept the hypothesis P(HA|D) based on 1-P(D|H0) on the grounds that, while they're not the same, they're proportional. (In that sense the p-value is like the poor man's likelihood ratio.) There's nothing wrong with p-values as a concept, but there's everything wrong with p-values in hypothesis testing. The misunderstanding is baked in.

Re: Psychology Journal Bans Significance Testing

#46

Post by thaumasiotes »

stdbrouw wrote: post #43
Al-Khwarizmi wrote: post #40

Earlier quoted context omitted.

I understand p-values, but I always have real problems understanding the thing of 95% confidence interval not meaning 95% probability of the true parameter being in the interval. I once grasped it, but then I forgot the reason. And now I look at this paragraph: "the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interv…

"The probability of this parameter being in the interval" is what's called a credible interval, and it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely. In general, frequentists just don't like to talk about the properties of this particular sample, only about long-term frequencies – hence the name. Why? Becau…

> it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely

Huh? Unless you can bound the set of potential results, this isn't possible. Say I want to estimate the half-life of some material (bounded below, but not above). A uniform prior doesn't exist. How will the credible interval relate to the confidence interval?

Re: Psychology Journal Bans Significance Testing

#47

Post by qznc »

NamTaf wrote: post #30
yummyfajitas wrote: post #28

Earlier quoted context omitted.

Not quite. The 5% represents the chance that if the null hypothesis is true , you would draw data at least as extreme as the data you just saw in a repeated experiment. Computing the probability that the data came from the theory stated in the null hypothesis would require a (Baysian) prior. Also, Tloewald's reply is completely and inexorably wrong. Tloewald seems to want a Bayesian answer, which frequentist statisti…

Thanks, that small distinction does make sense to me. I'm surprised I had it as close as I did.

Since I just learned yesterday that statisticians precisely distinguish between "probability" and "confidence", kudos for using it correctly. At least I believe you used them correctly.

Re: Psychology Journal Bans Significance Testing

#48

Post by im3w1l »

stdbrouw wrote: post #45
CountBayesie wrote: post #13

>The type of analysis being banned is often called a frequentist analysis I find that there is a trend of associating "bad statistics" with "Frequentists Statistics" which isn't really fair. If you found a statistician trained only in Frequentist methods and asked their opinion on experiment design in psychological research they would likely be just as appalled as any Bayesian. I'm a big fan of Bayesian methods, but…

The problem is that p-values are begging to be misunderstood, and in fact you cannot use them as a decisionmaking procedure without "misinterpreting" them – after all, you're deciding whether to accept the hypothesis P(HA|D) based on 1-P(D|H0) on the grounds that, while they're not the same, they're proportional. (In that sense the p-value is like the poor man's likelihood ratio.) There's nothing wrong with p-values…

You can update your posterior based on the p-values yourself though. "Well those eggheads may have disproved X, but X is just common sense, so I'm gonna keep believing it anyway. U-until I see more studies confirming the finding I mean."

Re: Psychology Journal Bans Significance Testing

#49

Post by [deleted] »

thaumasiotes wrote: post #46
stdbrouw wrote: post #43

Earlier quoted context omitted.

"The probability of this parameter being in the interval" is what's called a credible interval, and it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely. In general, frequentists just don't like to talk about the properties of this particular sample, only about long-term frequencies – hence the name. Why? Becau…

> it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely Huh? Unless you can bound the set of potential results, this isn't possible. Say I want to estimate the half-life of some material (bounded below, but not above). A uniform prior doesn't exist. How will the credible interval relate to the confidence interva…

[deleted]

Re: Psychology Journal Bans Significance Testing

#50

Post by Booktrope »

If the stats aren't based on samples of real-world data, rather than very rigorously designed experimental comparisons, I think there's another reason to be wary of p-values. [Note, in what follows I probably use some terminology wrong, because I'm not a statistician, but I do think the point is important and I don't see much written about it.] In the real world data is not a bunch of independent events, but events (or data points) that are interconnected in ways that are difficult to quantify. I once heard a presentation by an expert in AB testing at a leading tech company who became wary of his results and brought his concerns about non-independence of events that were being tested to corporate statisticians. (He was concerned that, even though the AB testing procedure supposedly randomized the test, interconnectedness within website traffic was not being accounted for.) By his account, the statisticians agreed it was a problem but recommended that he assume that variations due to non-independence would more or less balance each other out. He wasn't satisfied with this and said that he went ahead and did some more measurements, then ran out all the binomial expansions rather than relying on approximations. When he did this more detailed work, he found out that with at least some web-based AB tests, where conventional statistical formulas showed a p-value of .05, he was getting a confidence level of more like 30%. (I don't think you could call his measurement a p-value because he wasn't using the formulas normally used to compute p-values, but I think what he was saying was more or less that the p-value from the forumalae was .05 when it should have been .30 from a more rigorous look)

As I'm not a statistician by trade I don't keep up on the literature very well, but interconnectedness of data does seem to me to be a very important issue. I'm wondering if anyone can point me to some helpful reading to understand this side of the issue better. In particular, is there any approach to AB testing that can reliably address the issue of data interconnectedness in the kind of situation described above?

Post reply on HN