Live data from Hacker News

If you think psychological science is bad, imagine how bad it was in 1999

statmodeling.stat.columbia.edu

61–70 of 238 posts

Re: If you think psychological science is bad, imagine how bad it was in 1999

#61
post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

It isn't so much that it is not applied as that the pressure to generate data inside academic science makes it extremely difficult for grad students and postdocs to allocate the time to doing science on their experimental processes. They already only barely have time to do experiments on the system they are actually trying to study!

Engineering organizations inside major corporations usually actively engage in process improvement because they are resourced to do so.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#62
post #14

And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.

Somehow one of the 3 experimental groups has 16 people, whereas the other groups have 14 people. Wonder why...

Re: If you think psychological science is bad, imagine how bad it was in 1999

#63

If it is not a hard science, then it is likely a variant of philosophy.

What is a “hard science”?

The only distinction I care about is exactness and non-exactness.

Is the research based upon formulating a theory that is capable of forecassting not-yet observed events a nonexistent, exact margin or error, and are the conditions then re-created to see if the forecast is within the margin of error that the instruments that measure it have?

Some say biology is “hard”, and some say it is “soft”; some say many parts of cosmology are “hard” but they certainly aren't “exact”.

In exact science there are typically multiple ways to derive the same answer within one theory, and they all result into the exact same result.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#64
post #3

Psychological science is now thoroughly privatized. Don't read papers if this subject is of interest to your, look al big tech, ad tech, troll factories, influencer science, media consolidation, for-pay research-charities (great rabbithole to dive into btw). Academics are thoroughly out of the loop on this one, as are we all incidentally.

> for-pay research-charities (great rabbithole to dive into btw).

Any getting-started pointers for this (or your other suggestions)?

Re: If you think psychological science is bad, imagine how bad it was in 1999

#65

Much of such dubious "science" is packaged by self help authours as miracle principles which are going to set ones life right. Popular books like "Power Of Habit" and Cal Newport's books also seem to fall into this category, and seem to enjoy huge patronage, even in HN.

May you please explain what Cal Newport has stated , that is considered "dubious science."

Re: If you think psychological science is bad, imagine how bad it was in 1999

#66
post #34

Earlier quoted context omitted.

Honestly all sociology I have seen or been exposed to, including in college, seems to be more interested in acting as a platform to push specific ideas, rather than an attempt to find truth. Beyond that those involved in sociology seem to believe that a study is the same thing as an experiment and like to believe that constitutes proof. Ultimately we can't really run AB experiments on society at large because we are…

I don't disagree with you, but frankly would be a bit frustrating to limit one's studies of human behavior to just history without trying to understand the dynamics of current societies, trying to understand how they respond to change and so on. Both fields have a completely different set of instruments and very limited overlap.

The best social science studies involve often accidental experiments, where good experimental conditions occur not because of design but because of happenstance. The analysis of these situations could be construed as a historical case study, or it could be construed as an experiment. I agree that seeing analogues in past societies is not the best approach, but studying history can sometimes reveal experiment-like conditions.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#67
post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

Epistemology is an interesting field, but it's just *not part of the curriculum for engineering and experimental sciences types.

I only had a good introduction to it when I took it as an optional course in a humanities college.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#68
post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

"People" sciences like sociology, psychology and economics can make incredibly misleading claims because one experiment over a small sample of people at a certain moment in time might seem to support a claim, while the actual reason for the observed results is a factor which is never taken in consideration. On the other hand, conducting those experiments over wider demographics and in different points in time means t…

"On the other hand, conducting those experiments over wider demographics and in different points in time means that the study wants to build "universal" models of how each single person in the whole world acts, which is utterly dismissive of the specific local environment around people."

I don't think building "universal models" or observing recurring patterns through analysis of 'experiments over wider demographics and in different points in time' require the ambition to predict a single individual behavior or actions as a corollary.

The problem lies - like you said - with the policymaker. And well more generally with people who extrapolate the results of a paper inadequately.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#69
post #36
post #14

And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.

Which why P-scores need to be proportional to some nontrivial inverse factor of the number of experiments done in the field. Are there 10000 researchers doing 20000 experiments per year? Take 20000, multiply by the number of years we expect an academic to do hands on work (30?); invert: you need a P-score better than 1:600000.

This! Bonferroni correction across all conducted studies/analyses. I will suggest this next time I am reviewing a paper with crazy claims and shitty statistics.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#70
post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

I saw a chart once that made the point that most research studies can be classified along two axes: rigorousness of methods and popularity of results. Most published papers have either high rigor/low popularity and low rigor/high popularity. The trade-off is in the fact that highly rigorous studies only allow for narrow, unexciting results, while popular studies with flashy results will have to compromise on their rigorousness. This is not really a rule, but it is an interesting way to see research and the editorial/peer-review process.

The example discussed in OP seems to fall in the category of low rigor/high popularity. I am not 100% on my history of psych research, but it seems to me that the stereotype threat was all the rage in the late 90s following the publication of Steele and Aronson (1995). OP study seems to follow a similar experimental setup as S&A with a new group of people (Asian-American women).

As far as meta-science is concerned, I think that it remains mostly a part of philosophy (as in epistemology) and the focus of a few (senior?) scholars in each field. There is really no space to publish meta-scientific papers that "shake up" the field and call out established researchers, as editors that publish those pieces could come under similar criticism for their work. I think that it is not an accident that the discussion of the replication crisis in psychology started from blog posts and other non-academic avenues and then found its way to more "established" publications in the field (again, if I remember the context of those conversations).

I really wish that the review process was open. It would be interesting to see the reviewers comments to this specific paper and how the editor decided to pick up and engage with them. All those conversations are usually locked up in some editorial management system and are seldom made public. I don't know if we can really have open science without having open peer reviews.

Post reply on HN