Live data from Hacker News

We found only one-third of published psychology research is reliable – now what?

theconversation.com

71–74 of 74 posts

Re: We found only one-third of published psychology research is reliable – now what?

#71
post #7

One small suggestion: Eliminate "published and outa here," the policy where students have to publish their MS and Phd work in order to receive those degrees. All that does is to fill the journals with crap.

I agree in principle but oftentimes the research done by grad-students is part of a project that is spearheaded by a career researcher. I question the notion that MS and PhD work is filling journals with crap.

Instead, I think there are three problems:

1. The inherent problem of inference. When you set p 2. Certain (sub)disciplines of science aren't very scientific. In particular, they suffer from a high incidence of unfalsifiable claims and hand-wavy definitions. A good example, I think, is the shame vs guilt literature.

3. Certain (sub)disciplines of science are highly politicized. I point my finger at anything involving "identity" (in the sense of "identity politics"), "diversity" (sex, age, ethnicity, sexual orientation, etc) or "bias".

Regarding point 3, take a look at the faculty and students of any university you'd like and take stock of how many Black people are working on racism, women on sexism, LGBT on sexual orientation, etc. As a cognitive neuroscientist, I don't want to claim that people can't study themselves, but in light of the serious reproducibility issues that social psychology is facing, I think this correlation is anything but trivial.

I strongly suspect that the "hard psych vs social psych" dichotomy can be further broken down, and that the sub-disciplines listed in point 3 account for a disproportionate quantity of false-positive results, on accounts of flawed methodology. I suspect that much of this research is actually an attempt to validate political opinion.

I may be wrong about all of this, but at least the core claim testable with a handful of linear regressions.

Re: We found only one-third of published psychology research is reliable – now what?

#72
post #7

One small suggestion: Eliminate "published and outa here," the policy where students have to publish their MS and Phd work in order to receive those degrees. All that does is to fill the journals with crap.

I agree in principle but oftentimes the research done by grad-students is part of a project that is spearheaded by a career researcher. I question the notion that MS and PhD work is filling journals with crap. Instead, I think there are three problems: 1. The inherent problem of inference. When you set p 2. Certain (sub)disciplines of science aren't very scientific. In particular, they suffer from a high incidence of…

All good stuff. Thanks. I'm just saying, don't make publication a requirement for graduation.

I'm not a statistician, but as I understand it, a problem with the p value is that the standard calculation hinges on assumptions that are not necessarily met by the data, and if those assumptions were correctly accounted for, p would be higher.

If p < 0.05, and in practice 50% of published results are bunk, then regardless of the underlying math, a practical rule of thumb seems to be: Multiply p by 10.

Re: We found only one-third of published psychology research is reliable – now what?

#73

Earlier quoted context omitted.

I agree in principle but oftentimes the research done by grad-students is part of a project that is spearheaded by a career researcher. I question the notion that MS and PhD work is filling journals with crap. Instead, I think there are three problems: 1. The inherent problem of inference. When you set p 2. Certain (sub)disciplines of science aren't very scientific. In particular, they suffer from a high incidence of…

All good stuff. Thanks. I'm just saying, don't make publication a requirement for graduation. I'm not a statistician, but as I understand it, a problem with the p value is that the standard calculation hinges on assumptions that are not necessarily met by the data, and if those assumptions were correctly accounted for, p would be higher. If p < 0.05, and in practice 50% of published results are bunk, then regardless…

The practical rule of thumb for a p-value is that it's nothing more than the rate of expected false-positives. p=.05 means you can expect 5% of false positives.

If more than 5% of your studies are failing to replicate, then something else is involved, but it's not necessarily fraudulent. Post-hoc hypotheses are usually the biggest culprit, but it's insidious to the point that researchers often don't realize they're doing it.

Re: We found only one-third of published psychology research is reliable – now what?

#74
post #58

Earlier quoted context omitted.

Where did I say publication is a claim of conclusive proof? I said they published papers in which they claimed they conclusively proved something , and it turned out they didn't conclusively prove anything. Specifically, because their results couldn't be reproduced. In case you're not familiar with how experiments are carried out in natural sciences, "results couldn't be reproduced" means that 1. They claimed they go…

>Where did I say publication is a claim of conclusive proof? Exactly where you typed It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X" Again, this is patently false because they did not publish papers claiming conclusive proof. They published papers claiming evidence in favor of a theory. >Make up your mind already. There's no reason to be disrespectf…

> Returning to the point, it takes incredible mental gymnastics to argue that a false positive automatically degrades the status of a study from "scientific" to "unscientific"

No one said anything about A false positive!

"Cannot be reproduced" means there were a lot of false positives. So many, in fact, that you can't really draw any conclusion from the experiment. (Edit:) Or more to the point, that the p value the original authors claimed was bullshit.

Reproducing an experiment means reproducing both the experimental technique and the sample.

Post reply on HN