Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Psychology Journal Bans Significance Testing

sciencebasedmedicine.org

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Psychology Journal Bans Significance Testing

#71

Post by Gravityloss »

MollyR wrote: post #62

I might be missing the forest from the tree's. The paper talks about how it seems researches are "hacking" (their word) p-values.If the researchers lack the ethics using one form of statistics, what is really stopping them from misusing Bayesian Analysis? Sidenote: While we are talking about bayesian stuff. I recently ran into the sleeping beauty problem ( http://en.wikipedia.org/wiki/Sleeping_Beauty_problem ) and it…

That's a cool problem. It took a while to understand the angle.

I'd say I'm a halfer. I think the extreme sleeping beauty problem highlights how if you're woken up, it's not any more probable it's a head or a tails awakening, since, for a reason I can't explain, the million tails awakenings "don't accumulate".

In a sense, it's like the opposite of the Monty Hall problem, since here the sleeping beauty receives no information whatsoever during the experiment.

Re: Psychology Journal Bans Significance Testing

#72

Post by yummyfajitas »

Alex3917 wrote: post #67
yummyfajitas wrote: post #61

Earlier quoted context omitted.

[edit: I previously said I didn't understand Alex's point.] I now understand the point you were trying to make. A better way to put it - if a random number generator were used in a decision process, I'd favor making the algorithm and random seed explicit. Any procedure you use will have assumptions. You can't escape this. The only question is whether we show or hide them. Can you give an argument in favor of hidden a…

> Can you give an argument in favor of hidden assumptions and non-explicit procedures? So as counterintuitive as it sounds, I think there are actually a couple of good arguments that can be made here: 1) With significance testing, the burden of supplying the assumptions and determining meaning is largely on the reader. With bayesian, it's transferred to the author. While it might make sense to use Bayesian for things…

Regarding (1), with significance testing the burden of supplying assumptions is not placed on the reader. The assumptions are implicitly built into the NHST rather than explicitly built into the prior.

As for each study becoming a meta-study, that's silly. This is indeed the chewbacca defense. Rather, each empirical study provides Bayes factors which the reader can then use to update their posteriors.

Regarding (2), obviously not every number is a measurement. In Bayesian stats, numbers representing probabilities are quite explicitly opinions. They are meaningful on a ratio scale, and are even asymptotically known to be correct. But they aren't measurements.

(They are correct if your priors are absolutely continuous w.r.t. reality. If you hold a religious belief so strong that evidence can't change it ("100% certainty"), that's not an absolutely continuous prior.)

Re: Psychology Journal Bans Significance Testing

#73

Post by GhotiFish »

yummyfajitas wrote: post #28
NamTaf wrote: post #20

Earlier quoted context omitted.

Let me replay it to you and see if I understand it, because I don't know if I do: You assume a null hypothesis that (usually) represents the status quo of no influence between the theory and the data. You then collect data. The p value then describes the probability of that data aligning with / being as a result of the null hypothesis. In other words, a p Is that correct? I may have minced terms there because my stat…

Not quite. The 5% represents the chance that if the null hypothesis is true , you would draw data at least as extreme as the data you just saw in a repeated experiment. Computing the probability that the data came from the theory stated in the null hypothesis would require a (Baysian) prior. Also, Tloewald's reply is completely and inexorably wrong. Tloewald seems to want a Bayesian answer, which frequentist statisti…

could you explain the distinction between saying:

"Given the evidence, there is a >=5% probability of the null hypothesis being true"

and

"There is a >=5% probability that if the null hypothesis were true, that your data would be at least as extreme"

The only difference I see is how you avoided saying anything about the null hypothesis, but I don't see how you can avoid saying anything about it.

if the h0 were true, then the probability of the result is unlikely, how can you not conclude that h0 is unlikely? What step are you missing other than collecting a preponderance of evidence against it?

The article never enters into this distinction. It makes it clear that people misinterpret evidence against the null hypothesis as evidence of the alternative, which is a false dichotomy.

I am confused. I also have sympathy for Tloewald at this point.

Re: Psychology Journal Bans Significance Testing

#74

Post by MollyR »

Gravityloss wrote: post #71
MollyR wrote: post #62

I might be missing the forest from the tree's. The paper talks about how it seems researches are "hacking" (their word) p-values.If the researchers lack the ethics using one form of statistics, what is really stopping them from misusing Bayesian Analysis? Sidenote: While we are talking about bayesian stuff. I recently ran into the sleeping beauty problem ( http://en.wikipedia.org/wiki/Sleeping_Beauty_problem ) and it…

That's a cool problem. It took a while to understand the angle. I'd say I'm a halfer. I think the extreme sleeping beauty problem highlights how if you're woken up, it's not any more probable it's a head or a tails awakening, since, for a reason I can't explain, the million tails awakenings "don't accumulate". In a sense, it's like the opposite of the Monty Hall problem, since here the sleeping beauty receives no inf…

I'd say I'm a halfer too, but a lot of physicists/philosophers say 1/3, and it really bothers me that I can't seem to sync up with world leaders.

source : http://rfcwalters.blogspot.com/2014/08/the-sleeping-beauty-p...

Re: Psychology Journal Bans Significance Testing

#75

Post by Tloewald »

SpaceManNabs wrote: post #32
Tloewald wrote: post #25

Earlier quoted context omitted.

No there's a 5% chance the null hypothesis is correct. the probability your theory is correct is unknowable.

This is false. The null hypothesis is either correct or false. The veracity of the null hypothesis does not change as long as the experiments are repeated the same.

How does this contradict what I said? Enlighten me?

I said the theory -- by which I meant the non-null hypothesis -- is not endorsed by a low p value (indeed, this is a major point of the article). A low p-value says "hey they doesn't look like random data" not "your brilliant hypothesis is probably true". The data might not look random because of a methodological error, outright fraud, or a confound.

This is particularly important when you consider people looking at data over and over again trying to find "an effect". Theoretically, the tests are supposed to get tougher and tougher each time you examine the data (add one degree of freedom) but in practice this doesn't happen. It doesn't matter much with large data sets, but the social sciences often use datasets where n is roughly 100, and you might only have 20 subjects in a cell.

Re: Psychology Journal Bans Significance Testing

#76

Post by yummyfajitas »

GhotiFish wrote: post #73
yummyfajitas wrote: post #28

Earlier quoted context omitted.

Not quite. The 5% represents the chance that if the null hypothesis is true , you would draw data at least as extreme as the data you just saw in a repeated experiment. Computing the probability that the data came from the theory stated in the null hypothesis would require a (Baysian) prior. Also, Tloewald's reply is completely and inexorably wrong. Tloewald seems to want a Bayesian answer, which frequentist statisti…

could you explain the distinction between saying: "Given the evidence, there is a >=5% probability of the null hypothesis being true" and "There is a >=5% probability that if the null hypothesis were true, that your data would be at least as extreme" The only difference I see is how you avoided saying anything about the null hypothesis, but I don't see how you can avoid saying anything about it. if the h0 were true,…

Sure. Let H0 be the null hypothesis and D be the data you observed. The first statement is P(H0|D) = 0.05. The second is P(D|H0) = 0.05.

The two quantities are related to each other via Bayes rule:

P(H0|D)=P(D|H0)P(H0)/P(D)

So indeed, as P(D|H0) goes down, so does P(H0|D). But if P(H0)/P(D) is sufficiently large, you can easily have P(H0|D) high while P(D|H0) is low.

I too have sympathy for everyone confused by frequentist stats - they tend to answer the exact opposite question that one really wants answered. In contrast, Bayesian stats tend to answer the question that most people ask.

Re: Psychology Journal Bans Significance Testing

#77

Post by ejain »

WhitneyLand wrote: post #60
sunir wrote: post #55

Earlier quoted context omitted.

You have heard "19 times out of 20" described in the news? That is the 0.05 restated for laypeople. 1 time out of 20 you will get a false positive, in this case that the rabbit's foot worked.

Sorry maybe I'm being dense, but who would take 1 out of 20 success to mean they should start buying rabbit feet?

You'd start buying rabbit feet because only the study that was "successful" is published.

Re: Psychology Journal Bans Significance Testing

#78

Post by Booktrope »

stdbrouw wrote: post #70
Booktrope wrote: post #66

Earlier quoted context omitted.

The presenter I mentioned did not go into details about what interconnectedness he found, but I think it's quite obvious that people visiting a site do have influence on other visitors, which is at least a part of the underlying issue. On the simplest level, most web sites have share buttons to make it as easy as possible for visitors to influence other traffic. Or, other examples, a trending tweet can massively infl…

Hmmm, again, there are certainly ways that interactions between visitors can cause statistical dependence, but not in the specific case you mention. Let's take an A/B test on a referral funnel. If a user invites all of his friends, and his friends then visit the site, they will be randomized over A and B just like the original user, and so any effect that is not due to changes in the referral experience will simply n…

I understand if you think this is a non-issue, though I don't agree. The speaker I referenced about asked the statisticians at his company about this and they said it was a non-issue because things balanced out. He thought that was an idealization and claimed to have tested it building in some real world data, and reported that interconnected data of this kind drastically affected confidence levels. He didn't get into the details of how he measured interconnectedness, however.

The example you give seems to me to oversimplify the issue of complex interconnections between data points, as if the traffic on a real website came from one set of referrals, while in reality its much more complex, with referrers inducing other referrers and a variety of campaigns, postings, etc. influencing each other, and over time, overlaid in a fairly complex pattern. In other words, a bunch of interrelated data, very little of which is actually independent of other items.

I'm not really asking for an explanation of this in the comment thread here; what I'd like to know is, if there are any studies or other publications that deal with the issue of how to evaluate tests run on interconnected data of this kind.

Re: Psychology Journal Bans Significance Testing

#79

Post by stdbrouw »

Booktrope wrote: post #78
stdbrouw wrote: post #70

Earlier quoted context omitted.

Hmmm, again, there are certainly ways that interactions between visitors can cause statistical dependence, but not in the specific case you mention. Let's take an A/B test on a referral funnel. If a user invites all of his friends, and his friends then visit the site, they will be randomized over A and B just like the original user, and so any effect that is not due to changes in the referral experience will simply n…

I understand if you think this is a non-issue, though I don't agree. The speaker I referenced about asked the statisticians at his company about this and they said it was a non-issue because things balanced out. He thought that was an idealization and claimed to have tested it building in some real world data, and reported that interconnected data of this kind drastically affected confidence levels. He didn't get int…

There are absolutely ways to deal with what you call interconnected data, as I mentioned earlier: paired tests, corrections for autocorrelation, nonparametric and bootstrap methods for non-normal data and so on. But barring any examples of what you mean with interconnectedness in this context, it's hard to recommend any studies or publications because there is no One Method Of Interconnectedness Correction.

Also, statistics deals with many idealizations but the idea that randomization allows you to cleanly measure the effect of an intervention in the face of what would otherwise be confounding is simply not one of them. Sorry to disappoint, but with all you're telling us it simply sounds like the speaker was clueless.

Re: Psychology Journal Bans Significance Testing

#80

Post by mdbco »

esfandia wrote: post #37
mdbco wrote: post #7

The article is certainly correct that p-values and confidence intervals (or confidence sets, in multi-dimensional contexts) are widely misunderstood, not just in psychology or other social sciences, but in the hard sciences as well. The problem is even worse when you look outside of academia at common practices in more applied settings. As suggested, a good approach is to take p-values not as conclusive or decisive,…

Is the cautious approach then to treat a p-value in the absence of priors on the same level as a p-value in presence of unfavorable priors? When someone tests positive for a cancer test, the priors are known (probability of cancer in the general population is usually very low, and the false positive rate of the test may be relatively high), and so usually that first test is merely indication that further tests are ne…

> Is the cautious approach then to treat a p-value in the absence of priors on the same level as a p-value in presence of unfavorable priors?

In the presence of a poor prior the Bayesian probability would be biased in some way, so frequentists would say that the p-value in the absence of priors is actually superior in this case. Bayesians would reply that if they thought the prior might be poor then they would simply consider multiple different priors, but it's not clear how this would improve things much over the frequentist approach that simply assumes that the prior is unknown.

> So when you don't know the prior and you observe a low p-value on something, isn't that just "preliminary research" that needs to be further confirmed with other methods or at least the same test but using other data?

Yes, when you observe a p-value with low significance it should definitely indicate to you that more testing is necessary, either by using different testing methods, gathering new samples, or even just increasing the original sample size if that's possible. What I was trying to suggest in my last paragraph was that this should be the case even when we have highly significant p-values, because even significant p-values are not decisive. So even when we have "confirmatory research" that is highly statistically significant, we should still do all of the things that we would do when we have a p-value with low significance. It is sometimes the case that this subsequent research will overturn even very highly statistically significant results (though often this is unfortunately because mistakes in the original statistical methodology are uncovered).

Post reply on HN