Psychology Journal Bans Significance Testing
sciencebasedmedicine.org
Psychology Journal Bans Significance Testing
1–10 of 88 posts
Re: Psychology Journal Bans Significance Testing
#2There was a question yesterday about Evidence Based Medicine vs Science Based Medicine. The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help.
For example, EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice. The issue is that, as this article points out, even with a statistically significant result for such research, there's no plausible mechanism by which a rabbit's foot makes you lucky. Therefore the RCT is just one part of the whole picture, and subject to a specific type of manipulation.
Re: Psychology Journal Bans Significance Testing
#3There was a question yesterday about Evidence Based Medicine vs Science Based Medicine. The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. For example, EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice. The issue…
Something related to keep in mind when it comes to p-values is the false discovery rate:
> If you run lots of phase 2 trials with different drug candidates where only a minority (lets say 10%) actually work, then with standard trial statistics (80% power and 5% false positive rate) you will get 4.5% false positive and 8% true positives – so less than 2 out of every 3 positive trial results were real. A much lower success rate than the 5% error rate commonly assumed.
http://www.forbes.com/sites/davidgrainger/2015/01/29/why-too...
This recent (open access) review in R. Soc. Open Sci. the Forbes article was largely inspired by is worth a read too: http://rsos.royalsocietypublishing.org/content/1/3/140216
Re: Psychology Journal Bans Significance Testing
#4I wonder if it just boils down to this: Exclusive reliance on any single tool by an entire field for a long enough time period will eventually lead to a proliferation of bad results.
Re: Psychology Journal Bans Significance Testing
#5The audio is a bit quiet but the 'dance of the p values' video [1] linked to by the article is absolutely fantastic and well worth watching.
Re: Psychology Journal Bans Significance Testing
#6Good! This is a big step in improving everybody's work. If you aren't up on this research, here are the two articles that you need to read.
Why Most Published Research Findings are False: http://journals.plos.org/plosmedicine/article?id=10.1371/jou...
Revised Standards for Statistical Evidence: http://www.pnas.org/content/110/48/19313.short
Re: Psychology Journal Bans Significance Testing
#7The article is certainly correct that p-values and confidence intervals (or confidence sets, in multi-dimensional contexts) are widely misunderstood, not just in psychology or other social sciences, but in the hard sciences as well. The problem is even worse when you look outside of academia at common practices in more applied settings.
As suggested, a good approach is to take p-values not as conclusive or decisive, but rather as a tool that must be supplemented by other statistics. In particular, the article emphasizes Bayesian methods, which can certainly provide additional information, but this approach can also be rather limited when priors are not well-defined or are entirely unknown, which is unfortunately often the case in many problem domains.
One potential question is how to determine the nature of the distinction mentioned in the conclusion between "preliminary research" and "confirmatory research", particularly in cases where statistics provide the primary evidence, as in, e.g. psychology. Further studies in the same vein as the preliminary research can certainly provide additional supporting statistical evidence, but this doesn't escape the problem that all of the evidence is probabilistic in nature. The key issue here is that since statistical approaches can only give probabilistic evidence that a hypothesis is correct, then they strictly cannot tell you what is certainly true, so even confirmatory research is quite open to falsification. So we wouldn't want the label of "confirmatory research" to somehow suggest to the public the idea that it is certainly correct.
Re: Psychology Journal Bans Significance Testing
#8The problem is this: You need a solid background in statistics and some mathematics to perform these tests properly. Most apparently, people from that field lack these skills. If someone can't hit a nail on the head with a hammer, blaming and banning the use of hammers won't solve a thing.
Re: Psychology Journal Bans Significance Testing
#9I kind of which that the widely read publications, like newspapers, that have turned out science articles based on all these now pretty much discredited research articles would issue retractions or at least clarification for each of the articles that has been challenged in this regard.
Re: Psychology Journal Bans Significance Testing
#10The problem is this: You need a solid background in statistics and some mathematics to perform these tests properly. Most apparently, people from that field lack these skills. If someone can't hit a nail on the head with a hammer, blaming and banning the use of hammers won't solve a thing.
Taking hammers away will solve the problem of holes in things which aren't supposed to have them, including thumbs, walls, people's heads, and candy jars.