Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you think psychological science is bad, imagine how bad it was in 1999

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

151–160 of 238 posts

Re: If you think psychological science is bad, imagine how bad it was in 1999

#151

Post by ImprobableTruth »

akiselev wrote: post #133
ImprobableTruth wrote: post #106

Earlier quoted context omitted.

If there's "good social science" that can't be used to make predictions, what differentiates it from bad social science?

Its utility [1]. The social sciences study a lot of things that people in group A intuitively understand that group B can be completely ignorant of - say, for example, how to navigate a complex social structure like office politics in a modern workplace. Making any sort of predictions about intangible outcomes where the Hawthorne effect is in full effect is pretty much impossible since group A will respond to the new…

I don't understand what difference you're trying to make between utility and predictive power. If you can give information on what approach in general will be better to approach office politics that is just a prediction. It doesn't mean that these predictions have to be always right, but if they don't have predictive power and are no better than a coinflip, that "understanding" is just a post-rationalization that doesn't provide any utility at all.

At the very least, it seems to me like the person I originally responded to would also disagree with judging social sciences for its "utility" - the article they linked specifically contrasted it with the natural sciences that "solve problems".

Re: If you think psychological science is bad, imagine how bad it was in 1999

#152

Post by caenn »

fnord123 wrote: post #146
analog31 wrote: post #82

Earlier quoted context omitted.

Indeed, and if you think science is bad, imagine metascience.

When you think about it, it's insane that metametascience isn't studied more. We're going to question every little detail about the scientific process, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" metascience?

Indeed, and if you think metascience is bad, imagine meta-metascience.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#153

Post by yxwvut »

pytonslange wrote: post #69
thechao wrote: post #36

Earlier quoted context omitted.

Which why P-scores need to be proportional to some nontrivial inverse factor of the number of experiments done in the field. Are there 10000 researchers doing 20000 experiments per year? Take 20000, multiply by the number of years we expect an academic to do hands on work (30?); invert: you need a P-score better than 1:600000.

This! Bonferroni correction across all conducted studies/analyses. I will suggest this next time I am reviewing a paper with crazy claims and shitty statistics.

I know this is mostly a joke, but what you really want is Benjamini–Hochberg correction, unless you want to prevent even a single false discovery in all of science. FDR vs FWER

Re: If you think psychological science is bad, imagine how bad it was in 1999

#154

Post by fallingknife »

munificent wrote: post #24
m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

> it's insane that metascience isn't studied more. It is, probably more than is apparent. See: https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Sociology_of_scientific_knowle... There is an interesting recursive problem here, though: what tools do you use to scientifically analyze the scientific process? Whatever tools you use will themselves be hobbled by the same systemic flaws you…

> what tools do you use to scientifically analyze the scientific process?

Engineering. If you can build something that works based on the rules theorized by scientists, they are on to something. e.g. building a skyscraper proves we know the properties of steel to a pretty good margin of error.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#155

Post by analog31 »

caenn wrote: post #152
fnord123 wrote: post #146

Earlier quoted context omitted.

When you think about it, it's insane that metametascience isn't studied more. We're going to question every little detail about the scientific process, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" metascience?

Indeed, and if you think metascience is bad, imagine meta-metascience.

Where's tail call optimization when you need it?

Re: If you think psychological science is bad, imagine how bad it was in 1999

#156

Post by TaupeRanger »

denimnerd42 wrote: post #93
TaupeRanger wrote: post #12

Earlier quoted context omitted.

Same reason we believed eggs will give you heart disease. Science reporting is terrible and the general education system doesn't teach rational skepticism, it teaches unconditional trust of intellectual authority.

every mainstream thought about diet is untrustworthy to me due to the above type of example :/

Yes. The simple fact is that nutrition epidemiology has failed. We don't know much of anything, other than a handful of basics like avoiding loads of refined sugar and trans fats.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#157

Post by nicoburns »

munificent wrote: post #24
m12k wrote: post #15

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

> it's insane that metascience isn't studied more. It is, probably more than is apparent. See: https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Sociology_of_scientific_knowle... There is an interesting recursive problem here, though: what tools do you use to scientifically analyze the scientific process? Whatever tools you use will themselves be hobbled by the same systemic flaws you…

I might be biased as someone who studied philosophy. But I wish Philosophy of Science was mandatory in more science degrees. A lot of scientists don't seem to be familiar.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#158

Post by B-Con »

torginus wrote: post #149

What I don't understand is psychology research (at least in academia) does not seem to have moved behind the "we locked three dozen college kids in a classroom and had them perform some bizarre acts, through which we hope to pierce the veil of human nature" - style research. I feel like if something good could have come out of the social media age - is that we have documented the natural behaviour of a vast number of…

I've always been perplexed by this, too.

Aren't the two most important aspects of the research the data set and the study methodology? Why on earth would you skimp so heavily one of them?

I don't work in the sciences, but this kind of nonsense doesn't exist in the "actual" sciences. Physicists spend loads of money producing just the right experiment conditions and documenting the manner the experiment was created in. The dataset is incredibly important and very rigorously examined.

But in psych, the dataset is basically an after thought. "Oh by the way, we chose a small handful of kids who happened to be free at that time, with no reason to believe there's any geo, social, educational, political, or ethnic background diversity, it probably cost us like $200 plus some pizza. Now let's print the results in $5 million worth of textbooks for a few decades!"

I don't buy the funding argument. A professor probably costs the university 100-150k/yr and will be working on a small handful (2-6, ish?) of projects. Buying an hour of a subject's time for a study must cost, what, $30/hr? Shouldn't they be allocating a minimum of $50k in funding for the actual research, and dropping at least $10k for a good dataset?

I don't buy the argument that most experiments don't yield good results so the university is wary of funding them. At a minimum they should follow up a cheap test with promising results with a real experiment that has actual funding before everyone gets all excited about it.

Re: If you think psychological science is bad, imagine how bad it was in 1999

#159

Post by chromaton »

munificent wrote: post #136
chromaton wrote: post #32

Earlier quoted context omitted.

>what tools do you use to scientifically analyze the scientific process? Iterating on existing systems to see if you can get results to converge and also testing new systems to see if they also result in known good values.

It's fun to realize how deeply ingrained the current scientific process is in our way of thinking. All of these ideas that you tacitly take for granted are itself mutable parts of the scientific process: * That iterative improvement and hill-climbing is an effective process for improving results. * That replication of experiments and convergence is a truth-generating enterprise. * That truth can be expressed numerica…

Yeah, I'm reminded of Zen and the Art of Motorcycle maintenance where the author was driven to insanity by the quest to define "quality".

Re: If you think psychological science is bad, imagine how bad it was in 1999

#160

Post by PicassoCTs »

My 5 cents.

There is not one but two psychological sciences at the moment. One is public, publicly funded and in very bad shape, with most of its results being not reproduceable (reproduction crisis), and a partial destruction happening via neuro-sciences.

Who destroy, but do not offer large-scale replacement theories, that could encompass the whole species and are not in contradiction to other neuro-science results.

And then there is the second faction (Disclaimer: I can not proof what is deduced after this disclaimer.)

There are several cooperations and at least one government, which had the chance to large scale collect data on the population.

This data is a psychological gold-mine, if explored properly. One could query such a behavioral database and more important - enact virtual experiments.

Out of all male humans, who curse in front of the tv in the evening, filter out those who get into a car accident, plot the increase in cursing in front of the tv.

If taken to the extreme, this new, data-mining behavior sciences, could create a agent based model of the species in all variations and collect data only to check the expected outcome of a societal change against the real outcome, with spot samples.

I have my own little pet theories, how humanity would look to this privatized psychology, but i digress.

I think, academic psychology should have full access to all cooperation databases that contain behavioral data.

Post reply on HN