If you think psychological science is bad, imagine how bad it was in 1999 • Hacker News Board

Re: If you think psychological science is bad, imagine how bad it was in 1999

#11

Post by naasking » Wed, Jun 16, 2021, 3:21 PM UTC

Tarsul wrote: post #2

I still don't understand why we "believed" studies that claimed knowledge about typical behavior and subconscious decisions while those studies were only based on a single experiment with limited participants, e.g. Stanford Prison Experiment. These type of studies should ALWAYS be looked at critically and not just because they fail to reproduce but because they are based on a very small sample size in a very discreet…

The problem is not just with "belief", but in the process itself. Non-replicable studies are actually cited more than replicable studies:

https://advances.sciencemag.org/content/7/21/eabd1705

Science journalists probably are vulnerable to the same thing influences that lead scientists to do this, except they have even less review on their claims and so they become pop culture sound bites.

As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising, and per the above link, reviewers apply lower standards on these papers in the hopes of finding something truly interesting and/or exciting. Not just in the results mind you, sometimes papers also apply a novel methodology that might be worth wider discussion. I'm not sure that's worth the reduction in credibility though.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#12

Post by TaupeRanger » Wed, Jun 16, 2021, 3:21 PM UTC

Tarsul wrote: post #2

I still don't understand why we "believed" studies that claimed knowledge about typical behavior and subconscious decisions while those studies were only based on a single experiment with limited participants, e.g. Stanford Prison Experiment. These type of studies should ALWAYS be looked at critically and not just because they fail to reproduce but because they are based on a very small sample size in a very discreet…

Same reason we believed eggs will give you heart disease. Science reporting is terrible and the general education system doesn't teach rational skepticism, it teaches unconditional trust of intellectual authority.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#13

Post by burnte » Wed, Jun 16, 2021, 3:23 PM UTC

mistrial9 wrote: post #6

CactusOnFire wrote: post #4

Earlier quoted context omitted.

I became a Data Scientist because in 3rd year, I found the OKCupid data blog and realized their research had methodological soundness these "20 undergraduate lab students forced to be there" research never would. You trade 'shot-in-the-dark lab experiments' for 'a clear and obvious agenda'. The trick is to just make sure the agenda isn't morally reprehensible.

more than one third of the entire world population believes that OKCupid is "morally reprehensible" (!)

Religious biases don't count as valid moral objections. Religion has a strong negative effect on critical thinking skills (1, 2). If you dogs/pigs/shellfish/mixed-fabrics are inherently unclean or that women are property, your objections to a dating website are probably not going to have seriousl scientific ramifications.

1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5742220/ 2. https://www.google.com/search?q=Religion+has+a+strong+negati...

Re: If you think psychological science is bad, imagine how bad it was in 1999

#14

Post by tgv » Wed, Jun 16, 2021, 3:24 PM UTC

And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#15

Post by m12k » Wed, Jun 16, 2021, 3:27 PM UTC

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

Re: If you think psychological science is bad, imagine how bad it was in 1999

#16

Post by mykowebhn » Wed, Jun 16, 2021, 3:29 PM UTC

I worked at a well-known clinical psychology lab at an Ivy League school many years ago. There was so much "gaming" of the results.

For example, our PI didn't want to include a subject in our study because his scores weren't elevated enough, and our PI was worried that his score wouldn't drop enough which would adversely impact our results.

Another example: our active treatment therapists knew exactly what they were treating for, and our study was measuring improvements in the condition that was being treated. However, the control therapists had no idea what they were treating for, and we purposefully kept this information from them!

Re: If you think psychological science is bad, imagine how bad it was in 1999

#17

Post by mellavora » Wed, Jun 16, 2021, 3:32 PM UTC

burnte wrote: post #13

mistrial9 wrote: post #6

Earlier quoted context omitted.

more than one third of the entire world population believes that OKCupid is "morally reprehensible" (!)

Religious biases don't count as valid moral objections. Religion has a strong negative effect on critical thinking skills (1, 2). If you dogs/pigs/shellfish/mixed-fabrics are inherently unclean or that women are property, your objections to a dating website are probably not going to have seriousl scientific ramifications. 1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5742220/ 2. https://www.google.com/search?q=Reli…

Ok, I have some reason to agree with you, but if we rule out religion as a valid basis for morality, what do you suggest we replace it with?

Re: If you think psychological science is bad, imagine how bad it was in 1999

#18

Post by javert » Wed, Jun 16, 2021, 3:33 PM UTC

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

The incentive system that academic scientists live under explains why they don't push for studying metascience.

If that's going to happen, it has to come from outside the government-science complex.

Good luck with that...

Re: If you think psychological science is bad, imagine how bad it was in 1999

#19

Post by batguano » Wed, Jun 16, 2021, 3:34 PM UTC

Tarsul wrote: post #2

I still don't understand why we "believed" studies that claimed knowledge about typical behavior and subconscious decisions while those studies were only based on a single experiment with limited participants, e.g. Stanford Prison Experiment. These type of studies should ALWAYS be looked at critically and not just because they fail to reproduce but because they are based on a very small sample size in a very discreet…

I assume you are referring to "The Stanford Prison Anecdote"?

Re: If you think psychological science is bad, imagine how bad it was in 1999

#20

Post by ryanmarsh » Wed, Jun 16, 2021, 3:34 PM UTC

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

Similarly, I heard someone the other day assert that no one had done an double blind placebo controlled trial of the effects of FDA regulation.