Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you think psychological science is bad, imagine how bad it was in 1999

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

121–130 of 238 posts

Re: If you think psychological science is bad, imagine how bad it was in 1999

#121

Post by philwelch »

jedberg wrote: post #116

I studied CogSci in the late 90s, which involved taking psychology courses. Every psych course that ended in zero required you to participate in psychology experiments. The profs like to say that 18-22, mostly white/asian, mostly rich kids are the most studied group of people in the USA. Or course we now know that a ton of psychology research doesn't actually apply to people outside that small narrow window of people…

The social sciences have a term—“WEIRD” (Western, Educated, Industrialized, Rich and Democratic) to describe the population that most psychological and social science research has studied the most deeply. As the acronym implies, these populations are not normal across human history or even the modern world.

One of the fascinating concepts in abnormal psychology is the notion of a “culture-bound syndrome”—a mental illness that only occurs in a specific cultural milieu. I wonder how many mental illnesses are actually culture-bound syndromes of WEIRD culture?

Re: If you think psychological science is bad, imagine how bad it was in 1999

#122

Post by spullara »

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

Dedicated folks that just try and reproduce important papers would be amazing and valuable. Only time I have seen it happen at scale was during the Cold Fusion days.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#123

Post by jxramos »

m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

I was recently imagining an experiment classification tagging system: "trial", "reproduced", "peer reviewed", etc. I could imagine this set of information landing on some wikipedia page and the experiment in question would gain a bunch of these tag badges as understanding of the phenomenon matures.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#124

Post by derbOac »

tgv wrote: post #14

And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.

In the department I worked for as a faculty member, there was an issue where the other areas would have graduation rates around 40-50%, as opposed to our area which was above 95% or so. This came up in the context of an external review.

When people asked around, informally what was said was that the grad students in the other areas (especially one area, in the experimental molecular biosciences) would leave after having to "redo" their dissertation over and over again. Essentially what would happen is they would propose a dissertation study, it would be approved by the area committee, the student would do the study, and it would produce null results. So they would be told to redo it a different way, or to pick a different topic, it would get approved, and the same process would happen again. After this happened a few times, with the student being told they had to produce significant results, the student would grow despondent and leave the program.

What's sad about this is that it's formally reinforcing p-hacking basically, as part of the degree program. But it's even more absurd than what's often alluded to in meta-science writings, because in these cases you would have a formal graduate committee, composed of faculty, deciding that the dissertation thesis is a good one -- that the hypothesis and design are solid, and formally approving the dissertation proposal -- and then because the results are null, it's unacceptable. If this was being done so casually in that forum, I can't imagine what goes on behind the scenes.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#125

Post by throwawayboise »

spullara wrote: post #122
m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

Dedicated folks that just try and reproduce important papers would be amazing and valuable. Only time I have seen it happen at scale was during the Cold Fusion days.

That's because if cold fusion had been proven to work, there were fortunes to be made.

Most science has very little market value.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#126

Post by chromaton »

Tenoke wrote: post #79
chromaton wrote: post #62

Earlier quoted context omitted.

Somehow one of the 3 experimental groups has 16 people, whereas the other groups have 14 people. Wonder why...

For what is worth studies do end up with different amounts of people due to dropping out, people not matching criteria which are only checked later, indivisibility by number of groups etc.

Yes, there are legit reasons for this to happen, as well as cherry-picking.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#127

Post by derbOac »

programmarchy wrote: post #33
publicola1990 wrote: post #26

Much of such dubious "science" is packaged by self help authours as miracle principles which are going to set ones life right. Popular books like "Power Of Habit" and Cal Newport's books also seem to fall into this category, and seem to enjoy huge patronage, even in HN.

I'd bet that even more dubious science is packaged by large corporate interests. Take Big Tobacco, who had doctors saying smoking is healthy, or the sugar industry, corrupting guidelines and policy on coronary heart disease [1]. And medicine is not immune either, with financial conflicts of interest and pharmaceutical sponsorships correlating with outcomes. [2] [1] https://www.npr.org/sections/thetwo-way/2016/09/13/4…

I don't disagree with you, but might provocatively suggest that federal funding in the US vis-a-vis a large university could also be considered a large corporate interest at this point.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#128

Post by peoplefromibiza »

dash2 wrote: post #47
hpoe wrote: post #34

Earlier quoted context omitted.

Honestly all sociology I have seen or been exposed to, including in college, seems to be more interested in acting as a platform to push specific ideas, rather than an attempt to find truth. Beyond that those involved in sociology seem to believe that a study is the same thing as an experiment and like to believe that constitutes proof. Ultimately we can't really run AB experiments on society at large because we are…

I think you are naive about history. I'm an economist. If I threw away the half of the data that didn't support my findings, and got caught, I'd lose my job and never publish again. I'm pretty sure the same is true in other social sciences, such as psychology. This is true irrespective of the well-documented problems that the article describes, which certainly also apply in economics and elsewhere, to varying degrees…

But let's not pretend that historians influence the policy makers as much as sociologists and economists do either.

The damages when they are wrong are orders of magnitude bigger.

They are assumed to be right, sometimes even without proof, until they are tragically proven wrong.

And nobody lose their job anyway.

Have you ever seen a sociologist lose the job because proposed something to a politician that resulted in lots of people having their life ruined?

I never did, honestly.

Have the last three more recent economic and social crisis been caused by historians mistakes?

https://familyinequality.wordpress.com/2017/10/02/sociologys...

Re: If you think psychological science is bad, imagine how bad it was in 1999

#129

Post by cwkoss »

bluetomcat wrote: post #30
m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

"People" sciences like sociology, psychology and economics can make incredibly misleading claims because one experiment over a small sample of people at a certain moment in time might seem to support a claim, while the actual reason for the observed results is a factor which is never taken in consideration. On the other hand, conducting those experiments over wider demographics and in different points in time means t…

I majored in psychology in undergrad. A big part of why I didn't look for a psychology focused job is that the science is all so loose. I'd often learn about two different study-backed phenomena is two different classes that somewhat contradicted each other. Or I'd learn in a subsequent class that a previously taught study has been invalidated in one way or another. Almost everything is measured subjectively, so huge parts of our knowledge of psychology are a house of cards resting on assumptions that the diagnostic questionnaires used to measure are accurate and reliable. Many of the measured effects are small, and so it's hard to trust that randomization and controls are sufficient. Replication of results is a major issue.

It all just feels so 'loose' compared to the physical sciences.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#130

Post by bingidingi »

cwkoss wrote: post #129
bluetomcat wrote: post #30

Earlier quoted context omitted.

"People" sciences like sociology, psychology and economics can make incredibly misleading claims because one experiment over a small sample of people at a certain moment in time might seem to support a claim, while the actual reason for the observed results is a factor which is never taken in consideration. On the other hand, conducting those experiments over wider demographics and in different points in time means t…

I majored in psychology in undergrad. A big part of why I didn't look for a psychology focused job is that the science is all so loose. I'd often learn about two different study-backed phenomena is two different classes that somewhat contradicted each other. Or I'd learn in a subsequent class that a previously taught study has been invalidated in one way or another. Almost everything is measured subjectively, so huge…

Think about how loose medical science used to be (and for how long)! leeches, bloodletting, miasma, ridiculous enemas and all sorts of outright nonsense. We've got a lot more mistakes to make, but social sciences will improve too.
Post reply on HN