We found only one-third of published psychology research is reliable

Re: We found only one-third of published psychology research is reliable – now what?

#41

Post by omginternets » Fri, Nov 06, 2015, 3:19 PM UTC

weland wrote: post #36

omginternets wrote: post #19

Earlier quoted context omitted.

>This is a debate in semantics: research that is not successfully reproduced is not scientific. That's both untrue and fallacious (see: no true Scotsman ). If you set your p -value threshold at .05, then one in twenty experiments will produce a false positive. As such, plenty of research is conducted in a benevolent and meticulous fashion, only to yield a non-reproducible result. It's still scientific; it's just not…

> That's both untrue and fallacious (see: no true Scotsman). No it's not. A sound scientific approach requires that a theory be based on reproducible results. If an experiment that verifies your theory confirms your result today and infirms it tomorrow, then the theory, the experimental approach, or both, are wrong. Of course, experiments that can't be reproduced are part of the scientific endeavour. Every discovery…

>But treating them as anything other than stumbling steps that help you refine your understanding of the problem or as dead ends is as unscientific as it gets.

Which isn't at all what I'm suggesting.

I'm arguing against the idea that applying the scientific method and getting a false positive makes the effort unscientific.

So yes, it is both untrue and fallacious.

Re: We found only one-third of published psychology research is reliable – now what?

#42

Post by kyllo » Fri, Nov 06, 2015, 3:21 PM UTC

nonbel wrote: post #26

Now what? You need to attempt independent replications of every published claim going forward. It is clear a single published result is unreliable. This is well known but apparently needed to be rediscovered by those who misunderstand the meaning of a p-value. >"The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is…

You need to attempt independent replications of every published claim going forward.

Taking this opportunity to plug an open source application that I work on, for cataloging and verifying independent replications, called Curate Science: https://www.curatescience.org/

Re: We found only one-third of published psychology research is reliable – now what?

#43

Post by nonbel » Fri, Nov 06, 2015, 3:27 PM UTC

geomark wrote: post #38

nonbel wrote: post #33

Earlier quoted context omitted.

>"Let's be clear on the difference between reproducible and replication." I did not find this clear. What is the distinction you are trying to make?

Reproduce: take the published data and methods, rerun the analysis, see if you get the same results. No new experiment. Replicate: perform a new experiment with new subjects, apply the same methods, see if you get statistically equivalent results.

Thanks, I had not heard that distinction in terms before. I suppose it would be good to have, but I would not assume people are using them in that way.

Re: We found only one-third of published psychology research is reliable – now what?

#44

Post by omginternets » Fri, Nov 06, 2015, 3:27 PM UTC

Lawtonfogle wrote: post #28

omginternets wrote: post #6

I actually have an answer for this: Stop talking about "psychology" like it's a monolithic thing. Start by separating "hard" psychology (perception, attention, psychophysics, etc) from social psychology. Papers about saccadic adaptation are, I suspect, much more reproducible than those about implicit association.

One thing I noticed is that the more focused the subset of psychology, the better the results. For example psychology that looks at a group of neurons and how they function is vastly better than psychology looking at how some early life event X leads to later problems in life.

I don't think this is actually true. There are plenty of high-level effects that are very robust and reproducible: attentional cueing, perceptual masking, inhibition of return, saccadic adaptation, hyperbolic discounting, etc.

I think your hypothesis conflates a few notions, for instance hierarchical organization and study design. I think the relevant factor in your comparison is not the level of analysis, but the measure. "Early life event X" is likely to be measured by a survey, while studying neuronal populations is in the realm of electrophysiology.

Re: We found only one-third of published psychology research is reliable – now what?

#45

Post by omginternets » Fri, Nov 06, 2015, 3:29 PM UTC

drumdance wrote: post #31

Lawtonfogle wrote: post #28

Earlier quoted context omitted.

One thing I noticed is that the more focused the subset of psychology, the better the results. For example psychology that looks at a group of neurons and how they function is vastly better than psychology looking at how some early life event X leads to later problems in life.

Isn't that really neuroscience?

When neuroscience involves predicting behavior and/or mental state, it usually falls under the umbrella of "cognitive neuroscience", which in turn is a way of saying "psychology that's not soft as all hell".

The boundaries are fuzzy, and you'll find people with psych, medical, neuro, math, CS and philosophy degrees in experimental cogsci labs.

Re: We found only one-third of published psychology research is reliable – now what?

#46

Post by geomark » Fri, Nov 06, 2015, 3:30 PM UTC

nonbel wrote: post #26

Now what? You need to attempt independent replications of every published claim going forward. It is clear a single published result is unreliable. This is well known but apparently needed to be rediscovered by those who misunderstand the meaning of a p-value. >"The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is…

More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone doesn't tell the whole story.

[1] http://simplystatistics.org/2015/10/20/we-need-a-statistical...

Re: We found only one-third of published psychology research is reliable – now what?

#47

Post by chongli » Fri, Nov 06, 2015, 3:33 PM UTC

rjruxhchd wrote: post #21

Far more worrying is that only a third of economics findings can be replicated without the author's help, and yet these are the supposedly scientific findings our politicians insist on basing policy off of. Even if policy seems counterintuitive and clearly harmful to the general public, we're given the explanation "because economics says so." It's clear now from the Federal Reserve replicability study that we're bein…

Economists are the modern equivalent to the oracles at Delphi. They speak in gibberish and the priesthood (politicians) make the interpretations which coincidentally happen to favour themselves and their friends.

Re: We found only one-third of published psychology research is reliable – now what?

#48

Post by geomark » Fri, Nov 06, 2015, 3:36 PM UTC

nonbel wrote: post #43

geomark wrote: post #38

Earlier quoted context omitted.

Reproduce: take the published data and methods, rerun the analysis, see if you get the same results. No new experiment. Replicate: perform a new experiment with new subjects, apply the same methods, see if you get statistically equivalent results.

Thanks, I had not heard that distinction in terms before. I suppose it would be good to have, but I would not assume people are using them in that way.

Some people are making a lot of noise about making the distinction, like the guys teaching data science at John Hopkins. It's because both are important but they are distinctly different activities.

Re: We found only one-third of published psychology research is reliable – now what?

#49

Post by EdiX » Fri, Nov 06, 2015, 3:41 PM UTC

unprepare wrote: post #11

vox_mollis wrote: post #3

Sneaking suspicion that this will readily extend to other soft sciences, particularly sociology. A good chunk of our social and educational policy is informed by psychology and sociology. Since we have all grown up within various systems and institutions that have been largely informed by bad research, it's unlikely much will happen, since we see the claims made by these charlatans to be normal or common sense. Nonet…

This seems to me to be a clear cut case of poor incentives in the academic research sector. No graduate student is incentivized to re perform existing experiments to verify them, no they find much more success in creating a unique experiment which tests a previously untested hypothesis. This is modern academic science. If you aren't doing something newsworthy, you're not getting any funding.

>No graduate student is incentivized to re perform existing experiments to verify them

If you look at how this thing with psychology started ("replication bullies") or at what Brockman said about the LaCour scandal earlier this year (he didn't pursue publication initially because he thought he'd risk his career), it's not that they aren't incentivized, it's that they are actively discouraged.

Re: We found only one-third of published psychology research is reliable – now what?

#50

Post by DennisP » Fri, Nov 06, 2015, 3:51 PM UTC

geomark wrote: post #46

nonbel wrote: post #26

Now what? You need to attempt independent replications of every published claim going forward. It is clear a single published result is unreliable. This is well known but apparently needed to be rediscovered by those who misunderstand the meaning of a p-value. >"The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is…

More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone do…

So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.