Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Psychology Journal Bans Significance Testing

sciencebasedmedicine.org

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Psychology Journal Bans Significance Testing

#51

Post by nabla9 »

stdbrouw wrote: post #44

Banning p-values makes sense to me, as they force you to declare an effect as either significant or not significant, rather than looking at the preponderance of the evidence and building knowledge over multiple experiments. It also leads us to focus on statistical uncertainty at the expense of all of the other kinds of uncertainty researchers are faced with: do I have the theory to back this up, am I actually measuri…

Methodological flexibility in statistical modeling has gone trough the roof. New tools and estimation methods make Bayesian methods easy to use: Jags and Stan, Hamiltonian Monte Carlo, the variational Bayesian approximations, expectation propagation and even probabilistic programming (note: Bayesian in this context does not mean subjectivity. It's the ability to quantify uncertainty in the results and increased flexibility in the modeling).

New methods and tools are faster to use and give better results, but they require more statistical knowledge. If the scientist applying these methods or peer review can't understand the advances, it's all for nothing. Many sciences are methodologically very conservative to the extent that it holds the science back.

How do you increase the statistical knowledge of the field so much that peer review and researchers can be expected to understand and use the new methods if they can't be trusted with p-value?

Re: Psychology Journal Bans Significance Testing

#52

Post by stdbrouw »

Booktrope wrote: post #50

If the stats aren't based on samples of real-world data, rather than very rigorously designed experimental comparisons, I think there's another reason to be wary of p-values. [Note, in what follows I probably use some terminology wrong, because I'm not a statistician, but I do think the point is important and I don't see much written about it.] In the real world data is not a bunch of independent events, but events (…

Could you expand on what you mean by "interconnectedness within website traffic"?

If one person visiting the site has no influence on other people visiting the site, then measurements of their behavior will be independent. If Facebook tests a different interface on half of their users and the changed behavior of those users indirectly has some impact on the behavior of the control group, then your measurements would have some level of dependence – I can imagine that this could happen but it's not clear to me that this would be a common scenario. The same would happen if you measure the behavior of the same person more than once – but in this case there's many procedures for working with paired or autocorrelated data.

Re: Psychology Journal Bans Significance Testing

#54

Post by WhitneyLand »

phren0logy wrote: post #2

There was a question yesterday about Evidence Based Medicine vs Science Based Medicine. The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. For example, EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice. The issue…

But how could a randomized clinical trial make this mistake? What experimental design could make the rabbit's foot look good?

"EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice."

Re: Psychology Journal Bans Significance Testing

#55

Post by sunir »

WhitneyLand wrote: post #54
phren0logy wrote: post #2

There was a question yesterday about Evidence Based Medicine vs Science Based Medicine. The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. For example, EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice. The issue…

But how could a randomized clinical trial make this mistake? What experimental design could make the rabbit's foot look good? "EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice."

You have heard "19 times out of 20" described in the news? That is the 0.05 restated for laypeople. 1 time out of 20 you will get a false positive, in this case that the rabbit's foot worked.

Re: Psychology Journal Bans Significance Testing

#56

Post by bartkappenburg »

NamTaf wrote: post #20
rcthompson wrote: post #15

I spent about a hour explaining p-values to a fellow graduate-level researcher a few weeks ago. I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it. In undergrad, hypothesis testing was more or less taught as "do this inscrutable calculation and if the result is 0.05 or less, you win". The point is, in my experience, a lot of people really don't get p-values, e…

Let me replay it to you and see if I understand it, because I don't know if I do: You assume a null hypothesis that (usually) represents the status quo of no influence between the theory and the data. You then collect data. The p value then describes the probability of that data aligning with / being as a result of the null hypothesis. In other words, a p Is that correct? I may have minced terms there because my stat…

We were forced not to think in probabilities during my econometrics education if we talked about confidence intervals.

Say you have a confidence interval of 95% or higher (p value 95% of the variance within the data is explained by the model (you've come up with).

Re: Psychology Journal Bans Significance Testing

#57

Post by Alex3917 »

yummyfajitas wrote: post #31
Alex3917 wrote: post #11

Earlier quoted context omitted.

> The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. The strength of significance testing is that it purposely doesn't try to tell you how likely something is to be true, only how likely the data you got was the result of chance assuming the treatment is no better than placebo. You're still taking the pr…

You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. This is a danger sign - you are doing the same things the Bayesians do, just informally, less explicitly, and probably incorrectly. The fact is that to make a good decision, eventually you need to compute a single number. This is an elementary fact of topology: https://www.chrisstucchio.…

> That number will be based on some unproveable assumptions.

Given that, would you support using a random number generator as part of the drug approval process to remind people of the importance of the unknown and unknowable?

Re: Psychology Journal Bans Significance Testing

#58

Post by return0 »

i think this is also important on a conceptual level. There's just too much literature in life sciences nowadays which, instead of formulating bold hypotheses with clearly distinct effects, is happy to report miniscule effects that just crossed the arbitrary significance barrier.

Re: Psychology Journal Bans Significance Testing

#59

Post by nshepperd »

Al-Khwarizmi wrote: post #41
Al-Khwarizmi wrote: post #40

Earlier quoted context omitted.

I understand p-values, but I always have real problems understanding the thing of 95% confidence interval not meaning 95% probability of the true parameter being in the interval. I once grasped it, but then I forgot the reason. And now I look at this paragraph: "the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interv…

OK, I think I more or less got it and I can answer my own question... correct me if I'm wrong, but I think the interpretation "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?" is flawed because we are not talking about the same interval every time , right? If I have a variable that is alw…

IIRC, in bayesian terms, confidence intervals express `P(X ∈ [A,B] | X=x) = 0.95`, where X is the unknown parameter and [A,B] is the interval, assumed to be some fixed function of the data. Ordinarily I think they expect this to be satisfied for all values x. So this is P = 95% where the parameter is known but the interval is not (because the interval depends on the data, which is not known yet).

On the other hand credible intervals express `P(X ∈ [A, B] | A=a,B=b) = 0.95` (or more generally `P(X ∈ [a,b] | the data) = 0.95`). The latter is what is intuitively meant by "95% probability" of the true parameter being in the interval, because you do know a and b but not the parameter.

The example with random sampling of confidence intervals from {ℝ⁺, ℝ⁻} is indeed a good illustration of the difference.

Re: Psychology Journal Bans Significance Testing

#60

Post by WhitneyLand »

sunir wrote: post #55
WhitneyLand wrote: post #54

Earlier quoted context omitted.

But how could a randomized clinical trial make this mistake? What experimental design could make the rabbit's foot look good? "EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice."

You have heard "19 times out of 20" described in the news? That is the 0.05 restated for laypeople. 1 time out of 20 you will get a false positive, in this case that the rabbit's foot worked.

Sorry maybe I'm being dense, but who would take 1 out of 20 success to mean they should start buying rabbit feet?
Post reply on HN