If it is not a hard science, then it is likely a variant of philosophy.
If you think psychological science is bad, imagine how bad it was in 1999
31–40 of 238 posts
Re: If you think psychological science is bad, imagine how bad it was in 1999
#32When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?
> it's insane that metascience isn't studied more. It is, probably more than is apparent. See: https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Sociology_of_scientific_knowle... There is an interesting recursive problem here, though: what tools do you use to scientifically analyze the scientific process? Whatever tools you use will themselves be hobbled by the same systemic flaws you…
>what tools do you use to scientifically analyze the scientific process?
Iterating on existing systems to see if you can get results to converge and also testing new systems to see if they also result in known good values.
Re: If you think psychological science is bad, imagine how bad it was in 1999
#33Much of such dubious "science" is packaged by self help authours as miracle principles which are going to set ones life right. Popular books like "Power Of Habit" and Cal Newport's books also seem to fall into this category, and seem to enjoy huge patronage, even in HN.
I'd bet that even more dubious science is packaged by large corporate interests. Take Big Tobacco, who had doctors saying smoking is healthy, or the sugar industry, corrupting guidelines and policy on coronary heart disease [1]. And medicine is not immune either, with financial conflicts of interest and pharmaceutical sponsorships correlating with outcomes. [2]
[1] https://www.npr.org/sections/thetwo-way/2016/09/13/493739074...
Re: If you think psychological science is bad, imagine how bad it was in 1999
#34When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?
"People" sciences like sociology, psychology and economics can make incredibly misleading claims because one experiment over a small sample of people at a certain moment in time might seem to support a claim, while the actual reason for the observed results is a factor which is never taken in consideration. On the other hand, conducting those experiments over wider demographics and in different points in time means t…
Honestly all sociology I have seen or been exposed to, including in college, seems to be more interested in acting as a platform to push specific ideas, rather than an attempt to find truth.
Beyond that those involved in sociology seem to believe that a study is the same thing as an experiment and like to believe that constitutes proof.
Ultimately we can't really run AB experiments on society at large because we are living in; however humanity has at its disposal all of history as a case study. My point is if you really want to understand how societies interact and form, and react, and live ask a historian, not a sociologist.
I also would apply most of these comments to economics except there seems to be more diversity of viewpoints, and studies are used less than math to try and provide a veneer of respectability.
EDIT:
If someone feels that history is inferior to sociology for understanding how societies act and behave please tell me why. I want to understand where I am wrong. But I see a lot of our arguments that we are having in society nowadays the same as one's had a thousand years ago, the discussions over Social Media are basically the exact same ones people had over the printing press in Europe, I recently read "The Republic" and there were the exact same arguments I see repeated here.
So if you feel contrary please tell me why, I admit I could be wrong, but want to understand where my reasoning is flawed.
Re: If you think psychological science is bad, imagine how bad it was in 1999
#35I worked at a well-known clinical psychology lab at an Ivy League school many years ago. There was so much "gaming" of the results. For example, our PI didn't want to include a subject in our study because his scores weren't elevated enough, and our PI was worried that his score wouldn't drop enough which would adversely impact our results. Another example: our active treatment therapists knew exactly what they were…
Hence the “replication crisis”. Replication of experiment and differences in outcome highlight the “core” result. When the results are just totally different, obviously there was “gaming” or just a bad experiment.
Re: If you think psychological science is bad, imagine how bad it was in 1999
#36And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.
Which why P-scores need to be proportional to some nontrivial inverse factor of the number of experiments done in the field. Are there 10000 researchers doing 20000 experiments per year? Take 20000, multiply by the number of years we expect an academic to do hands on work (30?); invert: you need a P-score better than 1:600000.
Re: If you think psychological science is bad, imagine how bad it was in 1999
#37When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?
>When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science? In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns…
> In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.
I think this is a common critique, but I also think it is missing the point. What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those questions? [The alternative to good social science is not no social science, it’s bad social science](https://statmodeling.stat.columbia.edu/2021/03/12/the-social...).
Finance is also a bit tautological in this regard. It seems that often prediction models are impossible to disprove (e.g., our arbitrage method doesn't work anymore, the market updated). Yes good for putting skin in the game, but doesn't seem like it does much to advance our long-term understanding of humans.
Re: If you think psychological science is bad, imagine how bad it was in 1999
#38When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?
There is a ton of current and past scholarship on the sociology, history, and philosophy of science, and peer-review and publication is actually a pretty hot topic in those fields. Although yes, perhaps it would be nice if that work was better funded, or if practicing scientists paid more attention to it rather than just repeating old myths about how science works.
For example, here’s a scholarly article on the exact question you mention - how and where peer review came to be seen as a guarantor of scientific quality: https://www.journals.uchicago.edu/doi/abs/10.1086/700070 (tldr: it wasn’t the 17th century Royal Society; it’s much more recent.)
Re: If you think psychological science is bad, imagine how bad it was in 1999
#39And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.
I have a similar horror story.
A medical student worked hard to analyze, say, 40 x-rays out of hundreds available. He found no significant evidence for some hypothesis. When he told his supervisor, the reply was: "Well then you should just analyze some more x-rays. I'm sure you'll have a statistically significant result at some point."
Re: If you think psychological science is bad, imagine how bad it was in 1999
#40Actually I do not think that paper is that bad compared to majority (well I do not think Psychology currently is a science)