Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We found only one-third of published psychology research is reliable – now what?

theconversation.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: We found only one-third of published psychology research is reliable – now what?

#51

Post by weland »

omginternets wrote: post #41
weland wrote: post #36

Earlier quoted context omitted.

> That's both untrue and fallacious (see: no true Scotsman). No it's not. A sound scientific approach requires that a theory be based on reproducible results. If an experiment that verifies your theory confirms your result today and infirms it tomorrow, then the theory, the experimental approach, or both, are wrong. Of course, experiments that can't be reproduced are part of the scientific endeavour. Every discovery…

>But treating them as anything other than stumbling steps that help you refine your understanding of the problem or as dead ends is as unscientific as it gets. Which isn't at all what I'm suggesting. I'm arguing against the idea that applying the scientific method and getting a false positive makes the effort unscientific. So yes, it is both untrue and fallacious.

> Only a quarter of scientific drug research is successfully reproduced as well.[1]

The article you are mentioning in [1] refers to the fact that a quarter of the published drug research is not successfully reproduced.

That doesn't mean that people published papers saying "Hey, we did this experiment. Its results cannot be consistently reproduced, so we think it's unconclusive/because our theory is flawed with regards to this or that/because the experiment was flawed with regards to this or that and we think it can be refined by changing this approach or that apparatus".

It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X", but it turns out that their experiments cannot be consistently reproduced, so they're proof of exactly nothing.

That is unscientific.

Re: We found only one-third of published psychology research is reliable – now what?

#52

Post by [deleted] »

chongli wrote: post #47
rjruxhchd wrote: post #21

Far more worrying is that only a third of economics findings can be replicated without the author's help, and yet these are the supposedly scientific findings our politicians insist on basing policy off of. Even if policy seems counterintuitive and clearly harmful to the general public, we're given the explanation "because economics says so." It's clear now from the Federal Reserve replicability study that we're bein…

Economists are the modern equivalent to the oracles at Delphi. They speak in gibberish and the priesthood (politicians) make the interpretations which coincidentally happen to favour themselves and their friends.

[deleted]

Re: We found only one-third of published psychology research is reliable – now what?

#53

Post by nonbel »

DennisP wrote: post #50
geomark wrote: post #46

Earlier quoted context omitted.

More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone do…

So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.

>"there's a .04 probability that there's no effect and the results occurred by chance."

No, the p value is the probability of observing a result at least as extreme as your own given the null hypothesis is true. There are two errors here

1) Transposing the conditional: P(A|B) != P(B|A) http://rationalwiki.org/wiki/Confusion_of_the_inverse

2) Deviations from the null hypothesis can occur even the absence of a treatment effect, ie one of your model assumptions is wrong, baseline differences, etc.

Anyway, that is why instead of statistical significance researchers need to estimate the size of the effect. If you estimate the effect is in the range -1 to +2 then you publish that result. Others also estimate the range and see if these are consistent with each other.

Re: We found only one-third of published psychology research is reliable – now what?

#54

Post by nonbel »

geomark wrote: post #46
nonbel wrote: post #26

Now what? You need to attempt independent replications of every published claim going forward. It is clear a single published result is unreliable. This is well known but apparently needed to be rediscovered by those who misunderstand the meaning of a p-value. >"The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is…

More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone do…

From your link:

>"So should intersecting confidence intervals be our definition of replication? This too has a flaw since it favors imprecise studies with very large confidence intervals. If effect size is ignored, we may waste our time trying to replicate studies reporting practically meaningless findings."

I don't see why the definition of a replication should have anything to do with practical use or precision of the estimate. These are other important, but different, issues.

Intersecting estimates of the plausible range is fine as a definition. The observations are consistent with each other. The best way to make these estimates is yet another tangential issue.

Re: We found only one-third of published psychology research is reliable – now what?

#55

Post by omginternets »

weland wrote: post #51
omginternets wrote: post #41

Earlier quoted context omitted.

>But treating them as anything other than stumbling steps that help you refine your understanding of the problem or as dead ends is as unscientific as it gets. Which isn't at all what I'm suggesting. I'm arguing against the idea that applying the scientific method and getting a false positive makes the effort unscientific. So yes, it is both untrue and fallacious.

> Only a quarter of scientific drug research is successfully reproduced as well.[1] The article you are mentioning in [1] refers to the fact that a quarter of the published drug research is not successfully reproduced. That doesn't mean that people published papers saying "Hey, we did this experiment. Its results cannot be consistently reproduced, so we think it's unconclusive/because our theory is flawed with regard…

>It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X"

This is patently false. Publication is never a claim of conclusive proof; it's a claim of evidence.

I'm sorry, but you are wrong about this. False-positives don't suddenly make the experiment un-scientific. You're very misinformed about how science works:

- False positives are part of the landscape

- Contradictory evidence is part of the landscape

- The above issues are resolved by tracking reproducibility of results

You can come to a wrong conclusion using valid scientific means. The scientific method hinges on the assumption that research will eventually converge on a correct result.

Re: We found only one-third of published psychology research is reliable – now what?

#56

Post by nonbel »

sanoli wrote: post #25
subliminalzen wrote: post #13

It's not just psychological research that is deeply flawed. Only a quarter of scientific drug research is successfully reproduced as well.[1] Carl Jung claimed one of the chief factors responsible for mass brainwashing is scientific rationality.[2] Society worships the Goddess of Reason while frowning down on "irrational" and non-verifiable religious testimony. Now that science is proven to be systemically corrupt, w…

> Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on? Oh yes, and all the scientific advancements will stop working from now on since science is proven to be corrupt. I can hear the satellites and the MRI machines crashing because science doesn't work anymore... What this means is that studies are less rigorous than promoted to be.

You may be unaware that the idea and initial development of MRI came from someone who does not believe in the sodium potassium pump. In fact, it was developed specifically based on alternative theories regarding how ionic concentrations are regulated. He is also a creationist... https://en.wikipedia.org/wiki/Raymond_Vahan_Damadian

The guy who developed PCR (basically DNA testing) also has some interesting views: https://en.wikipedia.org/wiki/Kary_Mullis

From this I conclude that scientific advancement doesn't depend so much on commonly accepted scientific claims.

Re: We found only one-third of published psychology research is reliable – now what?

#57

Post by millermp12 »

When the science goes against the Narrative, do what our church fathers did: go with Scripture.

First, the American Anthropological Association, next the APA? They might as well come of the closet and stop pretending so I can stop putting dick quotes around social "scientists."

https://www.insidehighered.com/news/2010/11/30/anthroscience

Re: We found only one-third of published psychology research is reliable – now what?

#58

Post by weland »

omginternets wrote: post #55
weland wrote: post #51

Earlier quoted context omitted.

> Only a quarter of scientific drug research is successfully reproduced as well.[1] The article you are mentioning in [1] refers to the fact that a quarter of the published drug research is not successfully reproduced. That doesn't mean that people published papers saying "Hey, we did this experiment. Its results cannot be consistently reproduced, so we think it's unconclusive/because our theory is flawed with regard…

>It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X" This is patently false. Publication is never a claim of conclusive proof; it's a claim of evidence. I'm sorry, but you are wrong about this. False-positives don't suddenly make the experiment un-scientific. You're very misinformed about how science works: - False positives are part of the landscape -…

Where did I say publication is a claim of conclusive proof?

I said they published papers in which they claimed they conclusively proved something, and it turned out they didn't conclusively prove anything. Specifically, because their results couldn't be reproduced.

In case you're not familiar with how experiments are carried out in natural sciences, "results couldn't be reproduced" means that

1. They claimed they got with p

2. Some other guys repeated the same experiment ("repeated" as in they administered the same substances, to a sample of equal size under similar conditions and measured the same parameters under similar conditions) and it turned out that on their results, p was through the roof.

In some cases, that was simply because the authors didn't publish enough information for their experiments to be repeated (I was close to making that mistake, too. Thank God for review committees). But in most cases, that simply happened because authors cherry-picked data or "optimistically" interpreted results.

(Edit: Responsible review committees can sometimes spot the latter, but it's very hard to deal with the former. The correct thing to do is to have all researchers publish all their experimental data, even the one which wasn't included in the papers. A lot of researchers agree, but you'll find that a lot of companies that employ researchers actively invent reasons why their researchers shouldn't do that.)

> If you set your p-value threshold at .05, then one in twenty experiments will produce a false positive.

> The reason is simple: given a p-threshold of .05, one in five experiments will yield a false positive.

Make up your mind already.

Re: We found only one-third of published psychology research is reliable – now what?

#60

Post by DennisP »

nonbel wrote: post #53
DennisP wrote: post #50

Earlier quoted context omitted.

So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.

>"there's a .04 probability that there's no effect and the results occurred by chance." No, the p value is the probability of observing a result at least as extreme as your own given the null hypothesis is true. There are two errors here 1) Transposing the conditional: P(A|B) != P(B|A) http://rationalwiki.org/wiki/Confusion_of_the_inverse 2) Deviations from the null hypothesis can occur even the absence of a treatmen…

> probability of observing a result at least as extreme as your own given the null hypothesis is true.

My wording was poor but that's what I meant.

Post reply on HN