Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Psychology Journal Bans Significance Testing

sciencebasedmedicine.org

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Psychology Journal Bans Significance Testing

#81

Post by GhotiFish »

yummyfajitas wrote: post #76
GhotiFish wrote: post #73

Earlier quoted context omitted.

could you explain the distinction between saying: "Given the evidence, there is a >=5% probability of the null hypothesis being true" and "There is a >=5% probability that if the null hypothesis were true, that your data would be at least as extreme" The only difference I see is how you avoided saying anything about the null hypothesis, but I don't see how you can avoid saying anything about it. if the h0 were true,…

Sure. Let H0 be the null hypothesis and D be the data you observed. The first statement is P(H0|D) = 0.05. The second is P(D|H0) = 0.05. The two quantities are related to each other via Bayes rule: P(H0|D)=P(D|H0)P(H0)/P(D) So indeed, as P(D|H0) goes down, so does P(H0|D). But if P(H0)/P(D) is sufficiently large, you can easily have P(H0|D) high while P(D|H0) is low. I too have sympathy for everyone confused by frequ…

Could you clear up some notation for me?

What does P(D) mean?

I read that is, the probability of Data being true.

edit: to clear up my meaning.

I mean, it makes sense to me to ask "What is the probability of getting this data, given that the null hypothesis is true"

and "what is the probability of the null hypothesis being true, given this data"

but I don't know how "this data" evaluates on its own. I can't picture that

does it mean, how authoritative is the data? Maybe that's it.

edit: OK never mind I kinda worked it out on my own.

Re: Psychology Journal Bans Significance Testing

#82

Post by Paradigma11 »

SapphireSun wrote: post #35
NamTaf wrote: post #20

Earlier quoted context omitted.

Let me replay it to you and see if I understand it, because I don't know if I do: You assume a null hypothesis that (usually) represents the status quo of no influence between the theory and the data. You then collect data. The p value then describes the probability of that data aligning with / being as a result of the null hypothesis. In other words, a p Is that correct? I may have minced terms there because my stat…

Just to add to what the others say, rejecting the null hypothesis is also not evidence that your alternative hypothesis is correct. Often the hypothesis testing framework is stated something like: H0: µ = 0 (null hypothesis) Ha: µ ≠ 0 (alternative hypothesis) When you reject H0, it means that you can be somewhat confident that there was was some kind of distortion in your data that moved the mean away from (in this c…

But the additional theories that you will disprove in further experiments are a subset of Ha so how can disproving H0 not be seen as evidence that Ha is correct. How can H0: the 2 means are equal. and Ha: The 2 means are not equal not encompass the whole universe.

For practical purposes NHST is a function that returns either H0 or Ha.

Re: Psychology Journal Bans Significance Testing

#83

Post by Paradigma11 »

bartkappenburg wrote: post #56
NamTaf wrote: post #20

Earlier quoted context omitted.

Let me replay it to you and see if I understand it, because I don't know if I do: You assume a null hypothesis that (usually) represents the status quo of no influence between the theory and the data. You then collect data. The p value then describes the probability of that data aligning with / being as a result of the null hypothesis. In other words, a p Is that correct? I may have minced terms there because my stat…

We were forced not to think in probabilities during my econometrics education if we talked about confidence intervals. Say you have a confidence interval of 95% or higher (p value 95% of the variance within the data is explained by the model (you've come up with).

In my opinion the most useful answer would be, for example a mean: The mean is within this interval. We used a model that has this properties, 95% of the...

I think it is ridiculous that the main focus always seems to be arcane properties of the statistical algorithm and not the answer that it delivers.

Re: Psychology Journal Bans Significance Testing

#84

Post by Booktrope »

stdbrouw wrote: post #79
Booktrope wrote: post #78

Earlier quoted context omitted.

I understand if you think this is a non-issue, though I don't agree. The speaker I referenced about asked the statisticians at his company about this and they said it was a non-issue because things balanced out. He thought that was an idealization and claimed to have tested it building in some real world data, and reported that interconnected data of this kind drastically affected confidence levels. He didn't get int…

There are absolutely ways to deal with what you call interconnected data, as I mentioned earlier: paired tests, corrections for autocorrelation, nonparametric and bootstrap methods for non-normal data and so on. But barring any examples of what you mean with interconnectedness in this context, it's hard to recommend any studies or publications because there is no One Method Of Interconnectedness Correction. Also, sta…

Well, if he was clueless then two very large and successful tech companies had a clueless guy running their AB testing and showing great results in each context.

I'm certainly not looking for "One Method for Interconnectedness Correction" (especially not, as you put it, with each word capitalized). I'm looking for studies or papers that might have addressed anything like the effect of interconnectedness of web data on AB testing. I think you're saying, you don't know of any, and also that you personally don't think it's a real issue.

Re: Psychology Journal Bans Significance Testing

#85

Post by yummyfajitas »

GhotiFish wrote: post #81
yummyfajitas wrote: post #76

Earlier quoted context omitted.

Sure. Let H0 be the null hypothesis and D be the data you observed. The first statement is P(H0|D) = 0.05. The second is P(D|H0) = 0.05. The two quantities are related to each other via Bayes rule: P(H0|D)=P(D|H0)P(H0)/P(D) So indeed, as P(D|H0) goes down, so does P(H0|D). But if P(H0)/P(D) is sufficiently large, you can easily have P(H0|D) high while P(D|H0) is low. I too have sympathy for everyone confused by frequ…

Could you clear up some notation for me? What does P(D) mean? I read that is, the probability of Data being true. edit: to clear up my meaning. I mean, it makes sense to me to ask "What is the probability of getting this data, given that the null hypothesis is true" and "what is the probability of the null hypothesis being true, given this data" but I don't know how "this data" evaluates on its own. I can't picture t…

P(D) is the probability of observing the data you just saw, due to either the null or non-null hypothesis. It's a strictly Bayesian quantity, since it's dependent on a prior. If your model has only a null and alternative hypothesis, then:

    P(D) = P(D|H0)P(H0) + P(D|H1)P(H1)

Re: Psychology Journal Bans Significance Testing

#86

Post by christopheraden »

Al-Khwarizmi wrote: post #41
Al-Khwarizmi wrote: post #40

Earlier quoted context omitted.

I understand p-values, but I always have real problems understanding the thing of 95% confidence interval not meaning 95% probability of the true parameter being in the interval. I once grasped it, but then I forgot the reason. And now I look at this paragraph: "the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interv…

OK, I think I more or less got it and I can answer my own question... correct me if I'm wrong, but I think the interpretation "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?" is flawed because we are not talking about the same interval every time , right? If I have a variable that is alw…

The key thing to note is that the parameter being estimated is a fixed (but unknown) quantity. Unlike in Bayesian inference, where we assume the parameter (or our belief in it) is random. The distinction is important. We want inference about the parameter, yet because the interval is the random quantity with confidence intervals, we cannot assign probability statements to the parameter.

As a completely fabricated example, suppose the true proportion in a coin flip experiment is 40%. If my confidence interval is [.45, .65], what's the probability that .40 is in [.45, .65]? It's 1. The probability that a fixed, but unknown parameter will lie in any confidence interval will be either 0 (it isn't in the interval) or 1 (it is). The _proportion_ of times the interval contains the true parameter is the level of confidence (95%).

To your always-positive example, that procedure is not particularly weird. There's always a balancing act with CI's about length and confidence level (otherwise, I could choose all reals as my interval and get 100% confidence level). That your [0, inf) interval has 100% coverage means that you could probably shrink that interval so that it has finite upper bound without losing more than 5% confidence. Hard to say without a specific distribution in mind or mild assumptions, but an application of either Markov's or Chebyshev's Inequality would allow you to make really loose bounds with only relatively minor assumptions.

Re: Psychology Journal Bans Significance Testing

#87

Post by rcthompson »

NamTaf wrote: post #20
rcthompson wrote: post #15

I spent about a hour explaining p-values to a fellow graduate-level researcher a few weeks ago. I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it. In undergrad, hypothesis testing was more or less taught as "do this inscrutable calculation and if the result is 0.05 or less, you win". The point is, in my experience, a lot of people really don't get p-values, e…

Let me replay it to you and see if I understand it, because I don't know if I do: You assume a null hypothesis that (usually) represents the status quo of no influence between the theory and the data. You then collect data. The p value then describes the probability of that data aligning with / being as a result of the null hypothesis. In other words, a p Is that correct? I may have minced terms there because my stat…

Others have already explained it well enough, but I'll just add a few important concepts. First, if you remember nothing else, remember this: the p-value has absolutely nothing to do with the alternative hypothesis. In the context of p-values, there is no alternative hypothesis; it doesn't exist. The p-value only says anything about the relationship of your data to the null hypothesis. The second thing to keep in mind is that any null hypothesis likely has lots of assumptions built in to it, and if any of those is violated, the associated p-value is invalid. So you should always be aware of what those assumptions are, and try to verify that your data satisfied then if possible. (Almost no one does this, of course.) For example, the standard t-test assumes, among other things, that your data is normally distributed and that all groups have equal variance. If either of these assumptions is violated, the test is invalid.

Lastly, if you're in a situation where you're performing a large number of tests, always correct for multiple testing (i.e. false discovery rate or some similar method). Also, you can take advantage of your large data set to construct a negative control (e.g. by shuffling samples, the exact method will vary) and verify that your chosen statistical test gives a flat distribution of p-values, which is the expected result when all of the null hypotheses are true. If you have an excess of small p-values in the null data set, this indicates your test is producing false positives and is not reliable (presumably because one or more assumptions have been violated).

Re: Psychology Journal Bans Significance Testing

#88

Post by nzp »

chrisseaton wrote: post #53

I never understood where 0.05 came from. It seems like an arbitrary, magic number. Aren't those bad in science? Shouldn't we have some reason for every number we use? Why 0.05 instead of 0.049?

It is somewhat arbitrary. 0.05 corresponds to roughly 2, and 0.01 to roughly 3 standard deviations under the normal distribution. In high energy physics, for example, in the past a result with 3 sigma significance was considered “a discovery”, but it turned out there were a lot of false signals, so it was upped to 5 sigma (~ 3 x 10^{-7} percent chance of getting the effect assuming there is no effect in reality), while 3 sigma (or 4) result is considered “evidence of effect“. Unfortunately in softer sciences, considering their relatively generally poor state of research standards, we have no idea if 0.05 or 0.01 levels really are good standards of “discovery”.
Post reply on HN