Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We found only one-third of published psychology research is reliable – now what?

theconversation.com

Post reply on HN
Print view View original

61–70 of 74 posts

Re: We found only one-third of published psychology research is reliable – now what?

#61

Post by sanoli »

nonbel wrote: post #56
sanoli wrote: post #25

Earlier quoted context omitted.

> Now that science is proven to be systemically corrupt, what will "rational" people base their understanding on? Oh yes, and all the scientific advancements will stop working from now on since science is proven to be corrupt. I can hear the satellites and the MRI machines crashing because science doesn't work anymore... What this means is that studies are less rigorous than promoted to be.

You may be unaware that the idea and initial development of MRI came from someone who does not believe in the sodium potassium pump. In fact, it was developed specifically based on alternative theories regarding how ionic concentrations are regulated. He is also a creationist... https://en.wikipedia.org/wiki/Raymond_Vahan_Damadian The guy who developed PCR (basically DNA testing) also has some interesting views: http…

It doesn't matter. The people who develop newer MRI machines will use what they believe in the formulation of their experiments. If that initial development of MRI that you mention didn't amount to nothing, it would have been thrown aside by the medical and scientific communities. If I believe in the Loch Ness monster and I develop a new vaccine that can have its efficacy demonstrated in rigorous studies, so what about my Loch Ness? What part of him being a creationist affected the technical aspects of his MRI studies?

Re: We found only one-third of published psychology research is reliable – now what?

#62

Post by nonbel »

sanoli wrote: post #61
nonbel wrote: post #56

Earlier quoted context omitted.

You may be unaware that the idea and initial development of MRI came from someone who does not believe in the sodium potassium pump. In fact, it was developed specifically based on alternative theories regarding how ionic concentrations are regulated. He is also a creationist... https://en.wikipedia.org/wiki/Raymond_Vahan_Damadian The guy who developed PCR (basically DNA testing) also has some interesting views: http…

It doesn't matter. The people who develop newer MRI machines will use what they believe in the formulation of their experiments. If that initial development of MRI that you mention didn't amount to nothing, it would have been thrown aside by the medical and scientific communities. If I believe in the Loch Ness monster and I develop a new vaccine that can have its efficacy demonstrated in rigorous studies, so what abo…

>"What part of him being a creationist affected the technical aspects of his MRI studies?"

Nothing, I just gave those examples of people to show science can be systematically corrupt/wrong (since it is in their minds) and there will still be advancements.

Re: We found only one-third of published psychology research is reliable – now what?

#63

Post by nonbel »

DennisP wrote: post #60
nonbel wrote: post #53

Earlier quoted context omitted.

>"there's a .04 probability that there's no effect and the results occurred by chance." No, the p value is the probability of observing a result at least as extreme as your own given the null hypothesis is true. There are two errors here 1) Transposing the conditional: P(A|B) != P(B|A) http://rationalwiki.org/wiki/Confusion_of_the_inverse 2) Deviations from the null hypothesis can occur even the absence of a treatmen…

> probability of observing a result at least as extreme as your own given the null hypothesis is true. My wording was poor but that's what I meant.

They are two completely different things. Check out the link. Would you accidentally say "It's cloudy therefore there is a high probability of rain" when you meant "It's raining so there is a high probability of clouds"?

Re: We found only one-third of published psychology research is reliable – now what?

#64

Post by sanoli »

geomark wrote: post #38
nonbel wrote: post #33

Earlier quoted context omitted.

>"Let's be clear on the difference between reproducible and replication." I did not find this clear. What is the distinction you are trying to make?

Reproduce: take the published data and methods, rerun the analysis, see if you get the same results. No new experiment. Replicate: perform a new experiment with new subjects, apply the same methods, see if you get statistically equivalent results.

'Replicate' is clear to me. 'Reproduce', in this context, still isn't. Could you give an example?

Re: We found only one-third of published psychology research is reliable – now what?

#65

Post by gregn »

From 10 years of working as a sysadmin in a Psych department, one constant in life I've learned is that Psychologists are the most incompetent, fanciful egoists in all the sciences. They have no idea what is going on basically. Or, put as a friend of mine said, "People who go into Psychology are precisely those that do not intuitively understand it already." Implying that most of us have a decent, commonsense appraisal of how things work, but psych people are the most clueless of the clueless, that's why they go into a field (first falling for its false promises) that professes to explain life's mysteries to them. None of this is surprising, except perhaps that they scored so far away from the median.

Re: We found only one-third of published psychology research is reliable – now what?

#67

Post by sopooneo »

chongli wrote: post #47
rjruxhchd wrote: post #21

Far more worrying is that only a third of economics findings can be replicated without the author's help, and yet these are the supposedly scientific findings our politicians insist on basing policy off of. Even if policy seems counterintuitive and clearly harmful to the general public, we're given the explanation "because economics says so." It's clear now from the Federal Reserve replicability study that we're bein…

Economists are the modern equivalent to the oracles at Delphi. They speak in gibberish and the priesthood (politicians) make the interpretations which coincidentally happen to favour themselves and their friends.

I feel there's truth to that, but most at the edges. At core there is stuff economists tend to very widely agree on. There was a Planet Money episode outlining some pretty drastic policy changes that a panel of economists across the political spectrum all stood behind. Here's a write up: http://www.npr.org/sections/money/2012/07/19/157047211/six-p...

Re: We found only one-third of published psychology research is reliable – now what?

#68

Post by Asbostos »

unprepare wrote: post #11
vox_mollis wrote: post #3

Sneaking suspicion that this will readily extend to other soft sciences, particularly sociology. A good chunk of our social and educational policy is informed by psychology and sociology. Since we have all grown up within various systems and institutions that have been largely informed by bad research, it's unlikely much will happen, since we see the claims made by these charlatans to be normal or common sense. Nonet…

This seems to me to be a clear cut case of poor incentives in the academic research sector. No graduate student is incentivized to re perform existing experiments to verify them, no they find much more success in creating a unique experiment which tests a previously untested hypothesis. This is modern academic science. If you aren't doing something newsworthy, you're not getting any funding.

Not only is lack of replication bad, but I think also excess of new research is bad too. Everybody makes their own unique study, too different to compare to existing research, too small a sample size to be reliable, too singular among thousands of equally unimportant and unreliable studies that it goes unnoticed and unused.

Really, those researchers trying to do something novel but being incompetent at it, should put their skills to use replicating something else instead. Sure this would lead to them being seen as 2nd class scientists. But we still need that service, just as we still need cleaners even though they're seen as 2nd class workers.

Universities are the ones providing the incentives. Their management should be stopping and thinking about if their policies are adding value to scientific knowledge or subtracting it.

Re: We found only one-third of published psychology research is reliable – now what?

#69

Post by Asbostos »

DennisP wrote: post #12

A friend of mine is a social psychology researcher, who's published work destroying the statistics in other papers. He says this claim of unreliability is a lot more dubious than the media portrays. In many cases, they say a finding was not reproduced simply because the new study didn't quite meet the cutoff for "significance." However, if you do a meta-analysis of both studies, you find that the likelihood of the ef…

So we need the original authors to come back with their own re-replication. Instead of throwing their published paper over the wall and moving on, they are the people in the best position to actually defend it and address problems with replications.

This publish-and-forget behavior annoys me about researchers. Sometimes I find results that look like they'll useful for my (non-academic) work, but the paper containing them is the final say on the matter. There's no version 2, there's no other work improving on it. It's just a dead end.

Re: We found only one-third of published psychology research is reliable – now what?

#70

Post by omginternets »

weland wrote: post #58
omginternets wrote: post #55

Earlier quoted context omitted.

>It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X" This is patently false. Publication is never a claim of conclusive proof; it's a claim of evidence. I'm sorry, but you are wrong about this. False-positives don't suddenly make the experiment un-scientific. You're very misinformed about how science works: - False positives are part of the landscape -…

Where did I say publication is a claim of conclusive proof? I said they published papers in which they claimed they conclusively proved something , and it turned out they didn't conclusively prove anything. Specifically, because their results couldn't be reproduced. In case you're not familiar with how experiments are carried out in natural sciences, "results couldn't be reproduced" means that 1. They claimed they go…

>Where did I say publication is a claim of conclusive proof?

Exactly where you typed It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X"

Again, this is patently false because they did not publish papers claiming conclusive proof. They published papers claiming evidence in favor of a theory.

>Make up your mind already.

There's no reason to be disrespectful over a mistake. I meant 1 in 20 (5%, hence the mix-up).

Returning to the point, it takes incredible mental gymnastics to argue that a false positive automatically degrades the status of a study from "scientific" to "unscientific":

1. The adjective "scientific" describes a method, not a result. Those speaking of "scientific results" are either (a) referring to "results of a scientific study" or (b) confused about what science is (namely: a method, not a result).

2. A false positive degrades the status of a result (not a study) from "evidence in favor of X" to "not evidence in favor of X".

I must respectfully insist that you are wrong.

Post reply on HN